[==========] Running 5 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240501 20:00:02.797394 28519 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.217.254:42645
I20240501 20:00:02.798719 28519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:00:02.799755 28519 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:00:02.814815 28526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:02.814810 28528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:02.814877 28525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:03.923714 28527 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20240501 20:00:03.923863 28519 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20240501 20:00:03.927744 28519 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:00:03.927950 28519 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:00:03.928164 28519 hybrid_clock.cc:648] HybridClock initialized: now 1714593603928140 us; error 0 us; skew 500 ppm
I20240501 20:00:03.928974 28519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:00:03.935206 28519 webserver.cc:460] Webserver started at http://127.27.217.254:33109/ using document root <none> and password file <none>
I20240501 20:00:03.936038 28519 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:03.936290 28519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:00:03.936694 28519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:00:03.942620 28519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714593602638719-28519-0/minicluster-data/master-0-root/instance:
uuid: "6f1eb2bdd9334a72a8e3ff58db174edd"
format_stamp: "Formatted at 2024-05-01 20:00:03 on dist-test-slave-92m0"
I20240501 20:00:03.950129 28519 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20240501 20:00:03.955111 28534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:03.956218 28519 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:00:03.956559 28519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714593602638719-28519-0/minicluster-data/master-0-root
uuid: "6f1eb2bdd9334a72a8e3ff58db174edd"
format_stamp: "Formatted at 2024-05-01 20:00:03 on dist-test-slave-92m0"
I20240501 20:00:03.956904 28519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714593602638719-28519-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714593602638719-28519-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714593602638719-28519-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:04.019523 28519 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:04.021056 28519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:00:04.021489 28519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:04.088773 28519 rpc_server.cc:307] RPC server started. Bound to: 127.27.217.254:42645
I20240501 20:00:04.088857 28585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.217.254:42645 every 8 connection(s)
I20240501 20:00:04.096160 28586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:04.119004 28586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f1eb2bdd9334a72a8e3ff58db174edd: Bootstrap starting.
I20240501 20:00:04.124281 28586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f1eb2bdd9334a72a8e3ff58db174edd: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:04.125988 28586 log.cc:824] T 00000000000000000000000000000000 P 6f1eb2bdd9334a72a8e3ff58db174edd: Log is configured to *not* fsync() on all Append() calls
I20240501 20:00:04.130430 28586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f1eb2bdd9334a72a8e3ff58db174edd: No bootstrap required, opened a new log
I20240501 20:00:04.152319 28586 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f1eb2bdd9334a72a8e3ff58db174edd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f1eb2bdd9334a72a8e3ff58db174edd" member_type: VOTER }
I20240501 20:00:04.152854 28586 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f1eb2bdd9334a72a8e3ff58db174edd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:04.153059 28586 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f1eb2bdd9334a72a8e3ff58db174edd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f1eb2bdd9334a72a8e3ff58db174edd, State: Initialized, Role: FOLLOWER
I20240501 20:00:04.153750 28586 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f1eb2bdd9334a72a8e3ff58db174edd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f1eb2bdd9334a72a8e3ff58db174edd" member_type: VOTER }
I20240501 20:00:04.154218 28586 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f1eb2bdd9334a72a8e3ff58db174edd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:00:04.154443 28586 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f1eb2bdd9334a72a8e3ff58db174edd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:00:04.154703 28586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f1eb2bdd9334a72a8e3ff58db174edd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:04.158453 28586 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f1eb2bdd9334a72a8e3ff58db174edd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f1eb2bdd9334a72a8e3ff58db174edd" member_type: VOTER }
I20240501 20:00:04.159091 28586 leader_election.cc:302] T 00000000000000000000000000000000 P 6f1eb2bdd9334a72a8e3ff58db174edd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f1eb2bdd9334a72a8e3ff58db174edd; no voters:
I20240501 20:00:04.160720 28586 leader_election.cc:288] T 00000000000000000000000000000000 P 6f1eb2bdd9334a72a8e3ff58db174edd [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:00:04.161032 28589 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f1eb2bdd9334a72a8e3ff58db174edd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:00:04.163694 28589 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f1eb2bdd9334a72a8e3ff58db174edd [term 1 LEADER]: Becoming Leader. State: Replica: 6f1eb2bdd9334a72a8e3ff58db174edd, State: Running, Role: LEADER
I20240501 20:00:04.164674 28589 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f1eb2bdd9334a72a8e3ff58db174edd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f1eb2bdd9334a72a8e3ff58db174edd" member_type: VOTER }
I20240501 20:00:04.166163 28586 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f1eb2bdd9334a72a8e3ff58db174edd [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:00:04.176434 28590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f1eb2bdd9334a72a8e3ff58db174edd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f1eb2bdd9334a72a8e3ff58db174edd. Latest consensus state: current_term: 1 leader_uuid: "6f1eb2bdd9334a72a8e3ff58db174edd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f1eb2bdd9334a72a8e3ff58db174edd" member_type: VOTER } }
I20240501 20:00:04.177320 28590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f1eb2bdd9334a72a8e3ff58db174edd [sys.catalog]: This master's current role is: LEADER
I20240501 20:00:04.179661 28591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f1eb2bdd9334a72a8e3ff58db174edd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f1eb2bdd9334a72a8e3ff58db174edd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f1eb2bdd9334a72a8e3ff58db174edd" member_type: VOTER } }
I20240501 20:00:04.180307 28591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f1eb2bdd9334a72a8e3ff58db174edd [sys.catalog]: This master's current role is: LEADER
I20240501 20:00:04.180881 28599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:00:04.192143 28599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:00:04.205472 28519 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 20:00:04.213510 28599 catalog_manager.cc:1347] Generated new cluster ID: e16a3387618e4ea8aab7a1b84b2d239a
I20240501 20:00:04.213811 28599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:00:04.234661 28599 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:00:04.236833 28599 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:00:04.258185 28599 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 6f1eb2bdd9334a72a8e3ff58db174edd: Generated new TSK 0
I20240501 20:00:04.259347 28599 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:00:04.278216 28519 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:00:04.287334 28608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:04.288738 28609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:04.291699 28611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:04.292254 28519 server_base.cc:1034] running on GCE node
I20240501 20:00:04.293031 28519 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:00:04.293228 28519 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:00:04.293422 28519 hybrid_clock.cc:648] HybridClock initialized: now 1714593604293406 us; error 0 us; skew 500 ppm
I20240501 20:00:04.294054 28519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:00:04.298447 28519 webserver.cc:460] Webserver started at http://127.27.217.193:43317/ using document root <none> and password file <none>
I20240501 20:00:04.298964 28519 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:04.299156 28519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:00:04.299420 28519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:00:04.300565 28519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714593602638719-28519-0/minicluster-data/ts-0-root/instance:
uuid: "dabdbfb59e8645c297a1696630198b6e"
format_stamp: "Formatted at 2024-05-01 20:00:04 on dist-test-slave-92m0"
I20240501 20:00:04.306342 28519 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240501 20:00:04.310870 28616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:04.311694 28519 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 20:00:04.311977 28519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714593602638719-28519-0/minicluster-data/ts-0-root
uuid: "dabdbfb59e8645c297a1696630198b6e"
format_stamp: "Formatted at 2024-05-01 20:00:04 on dist-test-slave-92m0"
I20240501 20:00:04.312292 28519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714593602638719-28519-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714593602638719-28519-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714593602638719-28519-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:04.340080 28519 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:04.341372 28519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:04.344738 28519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:00:04.350334 28519 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:00:04.350661 28519 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:04.350986 28519 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:00:04.351217 28519 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:04.439679 28519 rpc_server.cc:307] RPC server started. Bound to: 127.27.217.193:46801
I20240501 20:00:04.439700 28678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.217.193:46801 every 8 connection(s)
I20240501 20:00:04.447549 28519 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:00:04.457645 28683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:04.458916 28684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:04.466933 28519 server_base.cc:1034] running on GCE node
W20240501 20:00:04.468123 28686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:04.469246 28519 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:00:04.469604 28519 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:00:04.469870 28519 hybrid_clock.cc:648] HybridClock initialized: now 1714593604469854 us; error 0 us; skew 500 ppm
I20240501 20:00:04.470748 28519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:00:04.474377 28519 webserver.cc:460] Webserver started at http://127.27.217.194:43103/ using document root <none> and password file <none>
I20240501 20:00:04.475068 28519 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:04.475319 28519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:00:04.475384 28679 heartbeater.cc:347] Connected to a master server at 127.27.217.254:42645
I20240501 20:00:04.475675 28519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:00:04.475852 28679 heartbeater.cc:464] Registering TS with master...
I20240501 20:00:04.476900 28679 heartbeater.cc:511] Master 127.27.217.254:42645 requested a full tablet report, sending...
I20240501 20:00:04.477241 28519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714593602638719-28519-0/minicluster-data/ts-1-root/instance:
uuid: "a068ef6770f14874bd1acb01bce2c329"
format_stamp: "Formatted at 2024-05-01 20:00:04 on dist-test-slave-92m0"
I20240501 20:00:04.479768 28551 ts_manager.cc:194] Registered new tserver with Master: dabdbfb59e8645c297a1696630198b6e (127.27.217.193:46801)
I20240501 20:00:04.481868 28551 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58496
I20240501 20:00:04.484334 28519 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.007s
I20240501 20:00:04.488292 28691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:04.489137 28519 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20240501 20:00:04.489396 28519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714593602638719-28519-0/minicluster-data/ts-1-root
uuid: "a068ef6770f14874bd1acb01bce2c329"
format_stamp: "Formatted at 2024-05-01 20:00:04 on dist-test-slave-92m0"
I20240501 20:00:04.489679 28519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714593602638719-28519-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714593602638719-28519-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714593602638719-28519-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:04.514909 28519 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:04.516220 28519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:04.518199 28519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:00:04.521035 28519 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:00:04.521240 28519 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:04.521459 28519 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:00:04.521631 28519 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:04.612018 28519 rpc_server.cc:307] RPC server started. Bound to: 127.27.217.194:37307
I20240501 20:00:04.612186 28753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.217.194:37307 every 8 connection(s)
I20240501 20:00:04.618706 28519 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:00:04.626768 28757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:04.629405 28758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:04.634443 28754 heartbeater.cc:347] Connected to a master server at 127.27.217.254:42645
I20240501 20:00:04.634841 28754 heartbeater.cc:464] Registering TS with master...
I20240501 20:00:04.635740 28754 heartbeater.cc:511] Master 127.27.217.254:42645 requested a full tablet report, sending...
I20240501 20:00:04.638015 28551 ts_manager.cc:194] Registered new tserver with Master: a068ef6770f14874bd1acb01bce2c329 (127.27.217.194:37307)
I20240501 20:00:04.638455 28519 server_base.cc:1034] running on GCE node
W20240501 20:00:04.638554 28761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:04.639516 28519 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:00:04.639727 28519 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:00:04.639772 28551 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58512
I20240501 20:00:04.639901 28519 hybrid_clock.cc:648] HybridClock initialized: now 1714593604639881 us; error 0 us; skew 500 ppm
I20240501 20:00:04.640595 28519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:00:04.643345 28519 webserver.cc:460] Webserver started at http://127.27.217.195:42659/ using document root <none> and password file <none>
I20240501 20:00:04.643797 28519 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:04.643985 28519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:00:04.644272 28519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:00:04.645345 28519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714593602638719-28519-0/minicluster-data/ts-2-root/instance:
uuid: "3d7668f070e74286b38fda9d898f1acb"
format_stamp: "Formatted at 2024-05-01 20:00:04 on dist-test-slave-92m0"
I20240501 20:00:04.649757 28519 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 20:00:04.652989 28765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:04.653771 28519 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 20:00:04.654029 28519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714593602638719-28519-0/minicluster-data/ts-2-root
uuid: "3d7668f070e74286b38fda9d898f1acb"
format_stamp: "Formatted at 2024-05-01 20:00:04 on dist-test-slave-92m0"
I20240501 20:00:04.654294 28519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714593602638719-28519-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714593602638719-28519-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1714593602638719-28519-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:04.665953 28519 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:04.667127 28519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:04.669193 28519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:00:04.671901 28519 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:00:04.672118 28519 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:04.672346 28519 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:00:04.672504 28519 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:04.759790 28519 rpc_server.cc:307] RPC server started. Bound to: 127.27.217.195:38723
I20240501 20:00:04.759887 28827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.217.195:38723 every 8 connection(s)
I20240501 20:00:04.774430 28828 heartbeater.cc:347] Connected to a master server at 127.27.217.254:42645
I20240501 20:00:04.774765 28828 heartbeater.cc:464] Registering TS with master...
I20240501 20:00:04.775439 28828 heartbeater.cc:511] Master 127.27.217.254:42645 requested a full tablet report, sending...
I20240501 20:00:04.777223 28551 ts_manager.cc:194] Registered new tserver with Master: 3d7668f070e74286b38fda9d898f1acb (127.27.217.195:38723)
I20240501 20:00:04.777480 28519 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013158049s
I20240501 20:00:04.778546 28551 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58516
I20240501 20:00:04.780586 28519 test_util.cc:268] Using random seed: -289455603
I20240501 20:00:04.830448 28551 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58528:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 20:00:04.833822 28551 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 20:00:04.945447 28644 tablet_service.cc:1467] Processing CreateTablet for tablet 2ddf397fabe8453b8e1601a863a1b84e (DEFAULT_TABLE table=test-workload [id=fae9e6da6eb84ba5816fd6d1e4b31a6f]), partition=RANGE (key) PARTITION VALUES < 238609294
I20240501 20:00:04.945864 28642 tablet_service.cc:1467] Processing CreateTablet for tablet 8a0820e7c49f40f89038e0af73567bac (DEFAULT_TABLE table=test-workload [id=fae9e6da6eb84ba5816fd6d1e4b31a6f]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20240501 20:00:04.947918 28639 tablet_service.cc:1467] Processing CreateTablet for tablet 101b282620ad41b28af298749f60ec52 (DEFAULT_TABLE table=test-workload [id=fae9e6da6eb84ba5816fd6d1e4b31a6f]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20240501 20:00:04.947970 28641 tablet_service.cc:1467] Processing CreateTablet for tablet 676f538543c2491399b66b7db1490445 (DEFAULT_TABLE table=test-workload [id=fae9e6da6eb84ba5816fd6d1e4b31a6f]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20240501 20:00:04.948446 28638 tablet_service.cc:1467] Processing CreateTablet for tablet 8bd1afda996d441b8d3eaa64dd36d1c7 (DEFAULT_TABLE table=test-workload [id=fae9e6da6eb84ba5816fd6d1e4b31a6f]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20240501 20:00:04.949954 28640 tablet_service.cc:1467] Processing CreateTablet for tablet 21dd29f1917a47f7ab496136a4c7ca07 (DEFAULT_TABLE table=test-workload [id=fae9e6da6eb84ba5816fd6d1e4b31a6f]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20240501 20:00:04.950317 28637 tablet_service.cc:1467] Processing CreateTablet for tablet fbcced7b903441278c70c4d3a1458be4 (DEFAULT_TABLE table=test-workload [id=fae9e6da6eb84ba5816fd6d1e4b31a6f]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20240501 20:00:04.951725 28643 tablet_service.cc:1467] Processing CreateTablet for tablet 6a175ef3d4c94d459208ea5cfc31e9d8 (DEFAULT_TABLE table=test-workload [id=fae9e6da6eb84ba5816fd6d1e4b31a6f]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20240501 20:00:04.952152 28636 tablet_service.cc:1467] Processing CreateTablet for tablet 14bfbbeb889d43218d5807488de4fed2 (DEFAULT_TABLE table=test-workload [id=fae9e6da6eb84ba5816fd6d1e4b31a6f]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20240501 20:00:04.953349 28644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ddf397fabe8453b8e1601a863a1b84e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:04.955935 28642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a0820e7c49f40f89038e0af73567bac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:04.956578 28640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21dd29f1917a47f7ab496136a4c7ca07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:04.957579 28641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 676f538543c2491399b66b7db1490445. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:04.958161 28639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 101b282620ad41b28af298749f60ec52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:04.958760 28643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a175ef3d4c94d459208ea5cfc31e9d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:04.967262 28638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bd1afda996d441b8d3eaa64dd36d1c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:04.972504 28636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14bfbbeb889d43218d5807488de4fed2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:04.973737 28637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbcced7b903441278c70c4d3a1458be4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:04.989609 28846 tablet_bootstrap.cc:492] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e: Bootstrap starting.
I20240501 20:00:04.993993 28846 tablet_bootstrap.cc:654] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:04.999230 28846 tablet_bootstrap.cc:492] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e: No bootstrap required, opened a new log
I20240501 20:00:04.999608 28846 ts_tablet_manager.cc:1367] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20240501 20:00:05.001462 28846 raft_consensus.cc:348] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.002022 28846 raft_consensus.cc:374] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:05.002331 28846 raft_consensus.cc:729] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dabdbfb59e8645c297a1696630198b6e, State: Initialized, Role: FOLLOWER
I20240501 20:00:05.003063 28846 consensus_queue.cc:260] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.005216 28719 tablet_service.cc:1467] Processing CreateTablet for tablet 2ddf397fabe8453b8e1601a863a1b84e (DEFAULT_TABLE table=test-workload [id=fae9e6da6eb84ba5816fd6d1e4b31a6f]), partition=RANGE (key) PARTITION VALUES < 238609294
I20240501 20:00:05.006557 28719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ddf397fabe8453b8e1601a863a1b84e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:05.007179 28717 tablet_service.cc:1467] Processing CreateTablet for tablet 8a0820e7c49f40f89038e0af73567bac (DEFAULT_TABLE table=test-workload [id=fae9e6da6eb84ba5816fd6d1e4b31a6f]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20240501 20:00:05.008924 28717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a0820e7c49f40f89038e0af73567bac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:05.011833 28679 heartbeater.cc:503] Master 127.27.217.254:42645 was elected leader, sending a full tablet report...
I20240501 20:00:05.012883 28718 tablet_service.cc:1467] Processing CreateTablet for tablet 6a175ef3d4c94d459208ea5cfc31e9d8 (DEFAULT_TABLE table=test-workload [id=fae9e6da6eb84ba5816fd6d1e4b31a6f]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20240501 20:00:05.014317 28846 ts_tablet_manager.cc:1398] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e: Time spent starting tablet: real 0.014s user 0.002s sys 0.011s
I20240501 20:00:05.024230 28718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a175ef3d4c94d459208ea5cfc31e9d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:05.025236 28716 tablet_service.cc:1467] Processing CreateTablet for tablet 676f538543c2491399b66b7db1490445 (DEFAULT_TABLE table=test-workload [id=fae9e6da6eb84ba5816fd6d1e4b31a6f]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20240501 20:00:05.030326 28715 tablet_service.cc:1467] Processing CreateTablet for tablet 21dd29f1917a47f7ab496136a4c7ca07 (DEFAULT_TABLE table=test-workload [id=fae9e6da6eb84ba5816fd6d1e4b31a6f]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20240501 20:00:05.025310 28713 tablet_service.cc:1467] Processing CreateTablet for tablet 8bd1afda996d441b8d3eaa64dd36d1c7 (DEFAULT_TABLE table=test-workload [id=fae9e6da6eb84ba5816fd6d1e4b31a6f]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20240501 20:00:05.034512 28715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21dd29f1917a47f7ab496136a4c7ca07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:05.039517 28712 tablet_service.cc:1467] Processing CreateTablet for tablet fbcced7b903441278c70c4d3a1458be4 (DEFAULT_TABLE table=test-workload [id=fae9e6da6eb84ba5816fd6d1e4b31a6f]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20240501 20:00:05.037772 28792 tablet_service.cc:1467] Processing CreateTablet for tablet 6a175ef3d4c94d459208ea5cfc31e9d8 (DEFAULT_TABLE table=test-workload [id=fae9e6da6eb84ba5816fd6d1e4b31a6f]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20240501 20:00:05.035116 28711 tablet_service.cc:1467] Processing CreateTablet for tablet 14bfbbeb889d43218d5807488de4fed2 (DEFAULT_TABLE table=test-workload [id=fae9e6da6eb84ba5816fd6d1e4b31a6f]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20240501 20:00:05.042865 28716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 676f538543c2491399b66b7db1490445. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:05.050292 28792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a175ef3d4c94d459208ea5cfc31e9d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:05.050339 28712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbcced7b903441278c70c4d3a1458be4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:05.042248 28714 tablet_service.cc:1467] Processing CreateTablet for tablet 101b282620ad41b28af298749f60ec52 (DEFAULT_TABLE table=test-workload [id=fae9e6da6eb84ba5816fd6d1e4b31a6f]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20240501 20:00:05.048810 28791 tablet_service.cc:1467] Processing CreateTablet for tablet 8a0820e7c49f40f89038e0af73567bac (DEFAULT_TABLE table=test-workload [id=fae9e6da6eb84ba5816fd6d1e4b31a6f]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20240501 20:00:05.054687 28791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a0820e7c49f40f89038e0af73567bac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:05.042649 28793 tablet_service.cc:1467] Processing CreateTablet for tablet 2ddf397fabe8453b8e1601a863a1b84e (DEFAULT_TABLE table=test-workload [id=fae9e6da6eb84ba5816fd6d1e4b31a6f]), partition=RANGE (key) PARTITION VALUES < 238609294
I20240501 20:00:05.048869 28790 tablet_service.cc:1467] Processing CreateTablet for tablet 676f538543c2491399b66b7db1490445 (DEFAULT_TABLE table=test-workload [id=fae9e6da6eb84ba5816fd6d1e4b31a6f]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20240501 20:00:05.065069 28846 tablet_bootstrap.cc:492] T 21dd29f1917a47f7ab496136a4c7ca07 P dabdbfb59e8645c297a1696630198b6e: Bootstrap starting.
I20240501 20:00:05.064968 28788 tablet_service.cc:1467] Processing CreateTablet for tablet 101b282620ad41b28af298749f60ec52 (DEFAULT_TABLE table=test-workload [id=fae9e6da6eb84ba5816fd6d1e4b31a6f]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20240501 20:00:05.067333 28711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14bfbbeb889d43218d5807488de4fed2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:05.069976 28789 tablet_service.cc:1467] Processing CreateTablet for tablet 21dd29f1917a47f7ab496136a4c7ca07 (DEFAULT_TABLE table=test-workload [id=fae9e6da6eb84ba5816fd6d1e4b31a6f]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20240501 20:00:05.067160 28787 tablet_service.cc:1467] Processing CreateTablet for tablet 8bd1afda996d441b8d3eaa64dd36d1c7 (DEFAULT_TABLE table=test-workload [id=fae9e6da6eb84ba5816fd6d1e4b31a6f]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20240501 20:00:05.074126 28793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ddf397fabe8453b8e1601a863a1b84e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:05.067860 28786 tablet_service.cc:1467] Processing CreateTablet for tablet fbcced7b903441278c70c4d3a1458be4 (DEFAULT_TABLE table=test-workload [id=fae9e6da6eb84ba5816fd6d1e4b31a6f]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20240501 20:00:05.081211 28790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 676f538543c2491399b66b7db1490445. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:05.084702 28785 tablet_service.cc:1467] Processing CreateTablet for tablet 14bfbbeb889d43218d5807488de4fed2 (DEFAULT_TABLE table=test-workload [id=fae9e6da6eb84ba5816fd6d1e4b31a6f]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20240501 20:00:05.087800 28788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 101b282620ad41b28af298749f60ec52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:05.084187 28713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bd1afda996d441b8d3eaa64dd36d1c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:05.095558 28785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14bfbbeb889d43218d5807488de4fed2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:05.101887 28787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bd1afda996d441b8d3eaa64dd36d1c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:05.103953 28786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbcced7b903441278c70c4d3a1458be4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:05.106462 28714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 101b282620ad41b28af298749f60ec52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:05.106498 28789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21dd29f1917a47f7ab496136a4c7ca07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:05.165445 28846 tablet_bootstrap.cc:654] T 21dd29f1917a47f7ab496136a4c7ca07 P dabdbfb59e8645c297a1696630198b6e: Neither blocks nor log segments found. Creating new log.
W20240501 20:00:05.183349 28598 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240501 20:00:05.186736 28849 tablet_bootstrap.cc:492] T 2ddf397fabe8453b8e1601a863a1b84e P a068ef6770f14874bd1acb01bce2c329: Bootstrap starting.
I20240501 20:00:05.192438 28848 raft_consensus.cc:482] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:05.193778 28848 raft_consensus.cc:504] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
W20240501 20:00:05.200790 28680 tablet.cc:2369] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:00:05.206008 28848 leader_election.cc:288] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a068ef6770f14874bd1acb01bce2c329 (127.27.217.194:37307), 3d7668f070e74286b38fda9d898f1acb (127.27.217.195:38723)
I20240501 20:00:05.220801 28850 tablet_bootstrap.cc:492] T 8a0820e7c49f40f89038e0af73567bac P 3d7668f070e74286b38fda9d898f1acb: Bootstrap starting.
I20240501 20:00:05.245553 28846 tablet_bootstrap.cc:492] T 21dd29f1917a47f7ab496136a4c7ca07 P dabdbfb59e8645c297a1696630198b6e: No bootstrap required, opened a new log
I20240501 20:00:05.249375 28846 ts_tablet_manager.cc:1367] T 21dd29f1917a47f7ab496136a4c7ca07 P dabdbfb59e8645c297a1696630198b6e: Time spent bootstrapping tablet: real 0.186s user 0.007s sys 0.010s
I20240501 20:00:05.249179 28849 tablet_bootstrap.cc:654] T 2ddf397fabe8453b8e1601a863a1b84e P a068ef6770f14874bd1acb01bce2c329: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:05.255625 28846 raft_consensus.cc:348] T 21dd29f1917a47f7ab496136a4c7ca07 P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.261313 28846 raft_consensus.cc:374] T 21dd29f1917a47f7ab496136a4c7ca07 P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:05.261758 28846 raft_consensus.cc:729] T 21dd29f1917a47f7ab496136a4c7ca07 P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dabdbfb59e8645c297a1696630198b6e, State: Initialized, Role: FOLLOWER
I20240501 20:00:05.263371 28846 consensus_queue.cc:260] T 21dd29f1917a47f7ab496136a4c7ca07 P dabdbfb59e8645c297a1696630198b6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.266191 28846 ts_tablet_manager.cc:1398] T 21dd29f1917a47f7ab496136a4c7ca07 P dabdbfb59e8645c297a1696630198b6e: Time spent starting tablet: real 0.016s user 0.006s sys 0.001s
I20240501 20:00:05.268261 28729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0820e7c49f40f89038e0af73567bac" candidate_uuid: "dabdbfb59e8645c297a1696630198b6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a068ef6770f14874bd1acb01bce2c329" is_pre_election: true
I20240501 20:00:05.269199 28846 tablet_bootstrap.cc:492] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e: Bootstrap starting.
I20240501 20:00:05.269384 28849 tablet_bootstrap.cc:492] T 2ddf397fabe8453b8e1601a863a1b84e P a068ef6770f14874bd1acb01bce2c329: No bootstrap required, opened a new log
I20240501 20:00:05.269074 28850 tablet_bootstrap.cc:654] T 8a0820e7c49f40f89038e0af73567bac P 3d7668f070e74286b38fda9d898f1acb: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:05.269884 28849 ts_tablet_manager.cc:1367] T 2ddf397fabe8453b8e1601a863a1b84e P a068ef6770f14874bd1acb01bce2c329: Time spent bootstrapping tablet: real 0.084s user 0.012s sys 0.004s
W20240501 20:00:05.272018 28617 leader_election.cc:341] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a068ef6770f14874bd1acb01bce2c329 (127.27.217.194:37307): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:00:05.273722 28849 raft_consensus.cc:348] T 2ddf397fabe8453b8e1601a863a1b84e P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } }
I20240501 20:00:05.276196 28849 raft_consensus.cc:374] T 2ddf397fabe8453b8e1601a863a1b84e P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:05.278954 28849 raft_consensus.cc:729] T 2ddf397fabe8453b8e1601a863a1b84e P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a068ef6770f14874bd1acb01bce2c329, State: Initialized, Role: FOLLOWER
I20240501 20:00:05.281013 28803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0820e7c49f40f89038e0af73567bac" candidate_uuid: "dabdbfb59e8645c297a1696630198b6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d7668f070e74286b38fda9d898f1acb" is_pre_election: true
I20240501 20:00:05.279691 28849 consensus_queue.cc:260] T 2ddf397fabe8453b8e1601a863a1b84e P a068ef6770f14874bd1acb01bce2c329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } }
I20240501 20:00:05.281723 28846 tablet_bootstrap.cc:654] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e: Neither blocks nor log segments found. Creating new log.
W20240501 20:00:05.283371 28617 leader_election.cc:341] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d7668f070e74286b38fda9d898f1acb (127.27.217.195:38723): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:00:05.285110 28617 leader_election.cc:302] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dabdbfb59e8645c297a1696630198b6e; no voters: 3d7668f070e74286b38fda9d898f1acb, a068ef6770f14874bd1acb01bce2c329
I20240501 20:00:05.285938 28848 raft_consensus.cc:2726] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:00:05.286149 28754 heartbeater.cc:503] Master 127.27.217.254:42645 was elected leader, sending a full tablet report...
I20240501 20:00:05.296455 28850 tablet_bootstrap.cc:492] T 8a0820e7c49f40f89038e0af73567bac P 3d7668f070e74286b38fda9d898f1acb: No bootstrap required, opened a new log
I20240501 20:00:05.296968 28850 ts_tablet_manager.cc:1367] T 8a0820e7c49f40f89038e0af73567bac P 3d7668f070e74286b38fda9d898f1acb: Time spent bootstrapping tablet: real 0.077s user 0.014s sys 0.007s
I20240501 20:00:05.299655 28850 raft_consensus.cc:348] T 8a0820e7c49f40f89038e0af73567bac P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.300462 28850 raft_consensus.cc:374] T 8a0820e7c49f40f89038e0af73567bac P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:05.300765 28850 raft_consensus.cc:729] T 8a0820e7c49f40f89038e0af73567bac P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d7668f070e74286b38fda9d898f1acb, State: Initialized, Role: FOLLOWER
I20240501 20:00:05.300940 28855 raft_consensus.cc:482] T 2ddf397fabe8453b8e1601a863a1b84e P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:05.301491 28850 consensus_queue.cc:260] T 8a0820e7c49f40f89038e0af73567bac P 3d7668f070e74286b38fda9d898f1acb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.285477 28849 ts_tablet_manager.cc:1398] T 2ddf397fabe8453b8e1601a863a1b84e P a068ef6770f14874bd1acb01bce2c329: Time spent starting tablet: real 0.015s user 0.008s sys 0.000s
I20240501 20:00:05.301687 28855 raft_consensus.cc:504] T 2ddf397fabe8453b8e1601a863a1b84e P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } }
I20240501 20:00:05.303807 28849 tablet_bootstrap.cc:492] T 8a0820e7c49f40f89038e0af73567bac P a068ef6770f14874bd1acb01bce2c329: Bootstrap starting.
I20240501 20:00:05.310537 28849 tablet_bootstrap.cc:654] T 8a0820e7c49f40f89038e0af73567bac P a068ef6770f14874bd1acb01bce2c329: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:05.312989 28828 heartbeater.cc:503] Master 127.27.217.254:42645 was elected leader, sending a full tablet report...
I20240501 20:00:05.313840 28846 tablet_bootstrap.cc:492] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e: No bootstrap required, opened a new log
I20240501 20:00:05.314316 28846 ts_tablet_manager.cc:1367] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e: Time spent bootstrapping tablet: real 0.047s user 0.011s sys 0.012s
I20240501 20:00:05.317426 28846 raft_consensus.cc:348] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } }
I20240501 20:00:05.318189 28846 raft_consensus.cc:374] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:05.318506 28846 raft_consensus.cc:729] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dabdbfb59e8645c297a1696630198b6e, State: Initialized, Role: FOLLOWER
I20240501 20:00:05.319177 28846 consensus_queue.cc:260] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } }
I20240501 20:00:05.323086 28850 ts_tablet_manager.cc:1398] T 8a0820e7c49f40f89038e0af73567bac P 3d7668f070e74286b38fda9d898f1acb: Time spent starting tablet: real 0.026s user 0.004s sys 0.003s
I20240501 20:00:05.323880 28855 leader_election.cc:288] T 2ddf397fabe8453b8e1601a863a1b84e P a068ef6770f14874bd1acb01bce2c329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d7668f070e74286b38fda9d898f1acb (127.27.217.195:38723), dabdbfb59e8645c297a1696630198b6e (127.27.217.193:46801)
I20240501 20:00:05.324149 28850 tablet_bootstrap.cc:492] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb: Bootstrap starting.
I20240501 20:00:05.333529 28850 tablet_bootstrap.cc:654] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:05.345364 28846 ts_tablet_manager.cc:1398] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e: Time spent starting tablet: real 0.031s user 0.001s sys 0.008s
I20240501 20:00:05.336434 28848 raft_consensus.cc:482] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:05.352382 28848 raft_consensus.cc:504] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } }
I20240501 20:00:05.354550 28848 leader_election.cc:288] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d7668f070e74286b38fda9d898f1acb (127.27.217.195:38723), a068ef6770f14874bd1acb01bce2c329 (127.27.217.194:37307)
I20240501 20:00:05.356802 28803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ddf397fabe8453b8e1601a863a1b84e" candidate_uuid: "dabdbfb59e8645c297a1696630198b6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d7668f070e74286b38fda9d898f1acb" is_pre_election: true
I20240501 20:00:05.357071 28846 tablet_bootstrap.cc:492] T 101b282620ad41b28af298749f60ec52 P dabdbfb59e8645c297a1696630198b6e: Bootstrap starting.
I20240501 20:00:05.352816 28849 tablet_bootstrap.cc:492] T 8a0820e7c49f40f89038e0af73567bac P a068ef6770f14874bd1acb01bce2c329: No bootstrap required, opened a new log
I20240501 20:00:05.363415 28849 ts_tablet_manager.cc:1367] T 8a0820e7c49f40f89038e0af73567bac P a068ef6770f14874bd1acb01bce2c329: Time spent bootstrapping tablet: real 0.060s user 0.015s sys 0.003s
W20240501 20:00:05.364037 28617 leader_election.cc:341] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d7668f070e74286b38fda9d898f1acb (127.27.217.195:38723): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:00:05.366819 28849 raft_consensus.cc:348] T 8a0820e7c49f40f89038e0af73567bac P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.367763 28849 raft_consensus.cc:374] T 8a0820e7c49f40f89038e0af73567bac P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:05.381039 28849 raft_consensus.cc:729] T 8a0820e7c49f40f89038e0af73567bac P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a068ef6770f14874bd1acb01bce2c329, State: Initialized, Role: FOLLOWER
I20240501 20:00:05.383941 28729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ddf397fabe8453b8e1601a863a1b84e" candidate_uuid: "dabdbfb59e8645c297a1696630198b6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a068ef6770f14874bd1acb01bce2c329" is_pre_election: true
I20240501 20:00:05.384541 28850 tablet_bootstrap.cc:492] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb: No bootstrap required, opened a new log
I20240501 20:00:05.384783 28846 tablet_bootstrap.cc:654] T 101b282620ad41b28af298749f60ec52 P dabdbfb59e8645c297a1696630198b6e: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:05.384709 28729 raft_consensus.cc:2454] T 2ddf397fabe8453b8e1601a863a1b84e P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dabdbfb59e8645c297a1696630198b6e in term 0.
I20240501 20:00:05.384979 28850 ts_tablet_manager.cc:1367] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb: Time spent bootstrapping tablet: real 0.061s user 0.003s sys 0.015s
I20240501 20:00:05.386027 28617 leader_election.cc:302] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a068ef6770f14874bd1acb01bce2c329, dabdbfb59e8645c297a1696630198b6e; no voters: 3d7668f070e74286b38fda9d898f1acb
I20240501 20:00:05.387320 28863 raft_consensus.cc:2781] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:00:05.387733 28863 raft_consensus.cc:482] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:00:05.387609 28850 raft_consensus.cc:348] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.388182 28863 raft_consensus.cc:3037] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:05.388264 28850 raft_consensus.cc:374] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:05.388964 28850 raft_consensus.cc:729] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d7668f070e74286b38fda9d898f1acb, State: Initialized, Role: FOLLOWER
I20240501 20:00:05.389637 28850 consensus_queue.cc:260] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.391745 28850 ts_tablet_manager.cc:1398] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20240501 20:00:05.392505 28849 consensus_queue.cc:260] T 8a0820e7c49f40f89038e0af73567bac P a068ef6770f14874bd1acb01bce2c329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.394814 28849 ts_tablet_manager.cc:1398] T 8a0820e7c49f40f89038e0af73567bac P a068ef6770f14874bd1acb01bce2c329: Time spent starting tablet: real 0.031s user 0.003s sys 0.004s
I20240501 20:00:05.397163 28849 tablet_bootstrap.cc:492] T 101b282620ad41b28af298749f60ec52 P a068ef6770f14874bd1acb01bce2c329: Bootstrap starting.
I20240501 20:00:05.399780 28850 tablet_bootstrap.cc:492] T 14bfbbeb889d43218d5807488de4fed2 P 3d7668f070e74286b38fda9d898f1acb: Bootstrap starting.
I20240501 20:00:05.410974 28850 tablet_bootstrap.cc:654] T 14bfbbeb889d43218d5807488de4fed2 P 3d7668f070e74286b38fda9d898f1acb: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:05.414345 28803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ddf397fabe8453b8e1601a863a1b84e" candidate_uuid: "a068ef6770f14874bd1acb01bce2c329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d7668f070e74286b38fda9d898f1acb" is_pre_election: true
W20240501 20:00:05.416371 28692 leader_election.cc:341] T 2ddf397fabe8453b8e1601a863a1b84e P a068ef6770f14874bd1acb01bce2c329 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d7668f070e74286b38fda9d898f1acb (127.27.217.195:38723): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:00:05.417318 28846 tablet_bootstrap.cc:492] T 101b282620ad41b28af298749f60ec52 P dabdbfb59e8645c297a1696630198b6e: No bootstrap required, opened a new log
I20240501 20:00:05.417762 28846 ts_tablet_manager.cc:1367] T 101b282620ad41b28af298749f60ec52 P dabdbfb59e8645c297a1696630198b6e: Time spent bootstrapping tablet: real 0.061s user 0.016s sys 0.007s
I20240501 20:00:05.420414 28846 raft_consensus.cc:348] T 101b282620ad41b28af298749f60ec52 P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.421356 28849 tablet_bootstrap.cc:654] T 101b282620ad41b28af298749f60ec52 P a068ef6770f14874bd1acb01bce2c329: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:05.421319 28846 raft_consensus.cc:374] T 101b282620ad41b28af298749f60ec52 P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:05.421833 28846 raft_consensus.cc:729] T 101b282620ad41b28af298749f60ec52 P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dabdbfb59e8645c297a1696630198b6e, State: Initialized, Role: FOLLOWER
I20240501 20:00:05.422399 28654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ddf397fabe8453b8e1601a863a1b84e" candidate_uuid: "a068ef6770f14874bd1acb01bce2c329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dabdbfb59e8645c297a1696630198b6e" is_pre_election: true
I20240501 20:00:05.422559 28846 consensus_queue.cc:260] T 101b282620ad41b28af298749f60ec52 P dabdbfb59e8645c297a1696630198b6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.419994 28863 raft_consensus.cc:504] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } }
I20240501 20:00:05.424342 28654 raft_consensus.cc:2379] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a068ef6770f14874bd1acb01bce2c329 in current term 1: Already voted for candidate dabdbfb59e8645c297a1696630198b6e in this term.
I20240501 20:00:05.427819 28694 leader_election.cc:302] T 2ddf397fabe8453b8e1601a863a1b84e P a068ef6770f14874bd1acb01bce2c329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a068ef6770f14874bd1acb01bce2c329; no voters: 3d7668f070e74286b38fda9d898f1acb, dabdbfb59e8645c297a1696630198b6e
I20240501 20:00:05.428998 28858 raft_consensus.cc:3037] T 2ddf397fabe8453b8e1601a863a1b84e P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:05.430779 28849 tablet_bootstrap.cc:492] T 101b282620ad41b28af298749f60ec52 P a068ef6770f14874bd1acb01bce2c329: No bootstrap required, opened a new log
I20240501 20:00:05.431058 28846 ts_tablet_manager.cc:1398] T 101b282620ad41b28af298749f60ec52 P dabdbfb59e8645c297a1696630198b6e: Time spent starting tablet: real 0.013s user 0.006s sys 0.001s
I20240501 20:00:05.431217 28849 ts_tablet_manager.cc:1367] T 101b282620ad41b28af298749f60ec52 P a068ef6770f14874bd1acb01bce2c329: Time spent bootstrapping tablet: real 0.034s user 0.008s sys 0.005s
I20240501 20:00:05.434063 28850 tablet_bootstrap.cc:492] T 14bfbbeb889d43218d5807488de4fed2 P 3d7668f070e74286b38fda9d898f1acb: No bootstrap required, opened a new log
I20240501 20:00:05.434501 28850 ts_tablet_manager.cc:1367] T 14bfbbeb889d43218d5807488de4fed2 P 3d7668f070e74286b38fda9d898f1acb: Time spent bootstrapping tablet: real 0.035s user 0.008s sys 0.003s
I20240501 20:00:05.434909 28849 raft_consensus.cc:348] T 101b282620ad41b28af298749f60ec52 P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.435683 28849 raft_consensus.cc:374] T 101b282620ad41b28af298749f60ec52 P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:05.435992 28849 raft_consensus.cc:729] T 101b282620ad41b28af298749f60ec52 P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a068ef6770f14874bd1acb01bce2c329, State: Initialized, Role: FOLLOWER
I20240501 20:00:05.436738 28849 consensus_queue.cc:260] T 101b282620ad41b28af298749f60ec52 P a068ef6770f14874bd1acb01bce2c329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.435842 28858 raft_consensus.cc:2726] T 2ddf397fabe8453b8e1601a863a1b84e P a068ef6770f14874bd1acb01bce2c329 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:00:05.437927 28846 tablet_bootstrap.cc:492] T 8bd1afda996d441b8d3eaa64dd36d1c7 P dabdbfb59e8645c297a1696630198b6e: Bootstrap starting.
I20240501 20:00:05.439199 28849 ts_tablet_manager.cc:1398] T 101b282620ad41b28af298749f60ec52 P a068ef6770f14874bd1acb01bce2c329: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20240501 20:00:05.437325 28850 raft_consensus.cc:348] T 14bfbbeb889d43218d5807488de4fed2 P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.439643 28850 raft_consensus.cc:374] T 14bfbbeb889d43218d5807488de4fed2 P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:05.439950 28850 raft_consensus.cc:729] T 14bfbbeb889d43218d5807488de4fed2 P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d7668f070e74286b38fda9d898f1acb, State: Initialized, Role: FOLLOWER
I20240501 20:00:05.440196 28849 tablet_bootstrap.cc:492] T 14bfbbeb889d43218d5807488de4fed2 P a068ef6770f14874bd1acb01bce2c329: Bootstrap starting.
I20240501 20:00:05.440280 28803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ddf397fabe8453b8e1601a863a1b84e" candidate_uuid: "dabdbfb59e8645c297a1696630198b6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d7668f070e74286b38fda9d898f1acb"
I20240501 20:00:05.440717 28850 consensus_queue.cc:260] T 14bfbbeb889d43218d5807488de4fed2 P 3d7668f070e74286b38fda9d898f1acb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
W20240501 20:00:05.441882 28617 leader_election.cc:341] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3d7668f070e74286b38fda9d898f1acb (127.27.217.195:38723): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:00:05.443013 28863 leader_election.cc:288] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e [CANDIDATE]: Term 1 election: Requested vote from peers 3d7668f070e74286b38fda9d898f1acb (127.27.217.195:38723), a068ef6770f14874bd1acb01bce2c329 (127.27.217.194:37307)
I20240501 20:00:05.443517 28859 raft_consensus.cc:482] T 8a0820e7c49f40f89038e0af73567bac P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:05.444025 28729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ddf397fabe8453b8e1601a863a1b84e" candidate_uuid: "dabdbfb59e8645c297a1696630198b6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a068ef6770f14874bd1acb01bce2c329"
I20240501 20:00:05.443934 28859 raft_consensus.cc:504] T 8a0820e7c49f40f89038e0af73567bac P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.446417 28859 leader_election.cc:288] T 8a0820e7c49f40f89038e0af73567bac P 3d7668f070e74286b38fda9d898f1acb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dabdbfb59e8645c297a1696630198b6e (127.27.217.193:46801), a068ef6770f14874bd1acb01bce2c329 (127.27.217.194:37307)
I20240501 20:00:05.447686 28850 ts_tablet_manager.cc:1398] T 14bfbbeb889d43218d5807488de4fed2 P 3d7668f070e74286b38fda9d898f1acb: Time spent starting tablet: real 0.013s user 0.000s sys 0.008s
I20240501 20:00:05.448446 28849 tablet_bootstrap.cc:654] T 14bfbbeb889d43218d5807488de4fed2 P a068ef6770f14874bd1acb01bce2c329: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:05.448626 28850 tablet_bootstrap.cc:492] T 2ddf397fabe8453b8e1601a863a1b84e P 3d7668f070e74286b38fda9d898f1acb: Bootstrap starting.
I20240501 20:00:05.454031 28846 tablet_bootstrap.cc:654] T 8bd1afda996d441b8d3eaa64dd36d1c7 P dabdbfb59e8645c297a1696630198b6e: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:05.459930 28729 raft_consensus.cc:2454] T 2ddf397fabe8453b8e1601a863a1b84e P a068ef6770f14874bd1acb01bce2c329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dabdbfb59e8645c297a1696630198b6e in term 1.
I20240501 20:00:05.465050 28617 leader_election.cc:302] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a068ef6770f14874bd1acb01bce2c329, dabdbfb59e8645c297a1696630198b6e; no voters: 3d7668f070e74286b38fda9d898f1acb
I20240501 20:00:05.469285 28850 tablet_bootstrap.cc:654] T 2ddf397fabe8453b8e1601a863a1b84e P 3d7668f070e74286b38fda9d898f1acb: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:05.470374 28863 raft_consensus.cc:2781] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:00:05.470858 28863 raft_consensus.cc:686] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e [term 1 LEADER]: Becoming Leader. State: Replica: dabdbfb59e8645c297a1696630198b6e, State: Running, Role: LEADER
I20240501 20:00:05.471639 28863 consensus_queue.cc:237] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } }
I20240501 20:00:05.476820 28846 tablet_bootstrap.cc:492] T 8bd1afda996d441b8d3eaa64dd36d1c7 P dabdbfb59e8645c297a1696630198b6e: No bootstrap required, opened a new log
I20240501 20:00:05.477380 28846 ts_tablet_manager.cc:1367] T 8bd1afda996d441b8d3eaa64dd36d1c7 P dabdbfb59e8645c297a1696630198b6e: Time spent bootstrapping tablet: real 0.040s user 0.012s sys 0.006s
I20240501 20:00:05.480405 28846 raft_consensus.cc:348] T 8bd1afda996d441b8d3eaa64dd36d1c7 P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.481096 28846 raft_consensus.cc:374] T 8bd1afda996d441b8d3eaa64dd36d1c7 P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:05.481377 28846 raft_consensus.cc:729] T 8bd1afda996d441b8d3eaa64dd36d1c7 P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dabdbfb59e8645c297a1696630198b6e, State: Initialized, Role: FOLLOWER
I20240501 20:00:05.482461 28846 consensus_queue.cc:260] T 8bd1afda996d441b8d3eaa64dd36d1c7 P dabdbfb59e8645c297a1696630198b6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.483047 28654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0820e7c49f40f89038e0af73567bac" candidate_uuid: "3d7668f070e74286b38fda9d898f1acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dabdbfb59e8645c297a1696630198b6e" is_pre_election: true
I20240501 20:00:05.484082 28850 tablet_bootstrap.cc:492] T 2ddf397fabe8453b8e1601a863a1b84e P 3d7668f070e74286b38fda9d898f1acb: No bootstrap required, opened a new log
I20240501 20:00:05.483840 28654 raft_consensus.cc:2454] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d7668f070e74286b38fda9d898f1acb in term 0.
I20240501 20:00:05.484973 28846 ts_tablet_manager.cc:1398] T 8bd1afda996d441b8d3eaa64dd36d1c7 P dabdbfb59e8645c297a1696630198b6e: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240501 20:00:05.484952 28850 ts_tablet_manager.cc:1367] T 2ddf397fabe8453b8e1601a863a1b84e P 3d7668f070e74286b38fda9d898f1acb: Time spent bootstrapping tablet: real 0.036s user 0.015s sys 0.000s
I20240501 20:00:05.485411 28768 leader_election.cc:302] T 8a0820e7c49f40f89038e0af73567bac P 3d7668f070e74286b38fda9d898f1acb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d7668f070e74286b38fda9d898f1acb, dabdbfb59e8645c297a1696630198b6e; no voters:
I20240501 20:00:05.485971 28846 tablet_bootstrap.cc:492] T 6a175ef3d4c94d459208ea5cfc31e9d8 P dabdbfb59e8645c297a1696630198b6e: Bootstrap starting.
I20240501 20:00:05.486181 28859 raft_consensus.cc:2781] T 8a0820e7c49f40f89038e0af73567bac P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:00:05.486522 28859 raft_consensus.cc:482] T 8a0820e7c49f40f89038e0af73567bac P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:00:05.486845 28859 raft_consensus.cc:3037] T 8a0820e7c49f40f89038e0af73567bac P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:05.491685 28849 tablet_bootstrap.cc:492] T 14bfbbeb889d43218d5807488de4fed2 P a068ef6770f14874bd1acb01bce2c329: No bootstrap required, opened a new log
I20240501 20:00:05.492138 28849 ts_tablet_manager.cc:1367] T 14bfbbeb889d43218d5807488de4fed2 P a068ef6770f14874bd1acb01bce2c329: Time spent bootstrapping tablet: real 0.052s user 0.011s sys 0.010s
I20240501 20:00:05.493198 28859 raft_consensus.cc:504] T 8a0820e7c49f40f89038e0af73567bac P 3d7668f070e74286b38fda9d898f1acb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.494949 28859 leader_election.cc:288] T 8a0820e7c49f40f89038e0af73567bac P 3d7668f070e74286b38fda9d898f1acb [CANDIDATE]: Term 1 election: Requested vote from peers dabdbfb59e8645c297a1696630198b6e (127.27.217.193:46801), a068ef6770f14874bd1acb01bce2c329 (127.27.217.194:37307)
I20240501 20:00:05.494642 28849 raft_consensus.cc:348] T 14bfbbeb889d43218d5807488de4fed2 P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.495349 28849 raft_consensus.cc:374] T 14bfbbeb889d43218d5807488de4fed2 P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:05.495633 28849 raft_consensus.cc:729] T 14bfbbeb889d43218d5807488de4fed2 P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a068ef6770f14874bd1acb01bce2c329, State: Initialized, Role: FOLLOWER
I20240501 20:00:05.495837 28654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0820e7c49f40f89038e0af73567bac" candidate_uuid: "3d7668f070e74286b38fda9d898f1acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dabdbfb59e8645c297a1696630198b6e"
I20240501 20:00:05.496443 28654 raft_consensus.cc:3037] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:05.496308 28849 consensus_queue.cc:260] T 14bfbbeb889d43218d5807488de4fed2 P a068ef6770f14874bd1acb01bce2c329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.497048 28850 raft_consensus.cc:348] T 2ddf397fabe8453b8e1601a863a1b84e P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } }
I20240501 20:00:05.499245 28849 ts_tablet_manager.cc:1398] T 14bfbbeb889d43218d5807488de4fed2 P a068ef6770f14874bd1acb01bce2c329: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240501 20:00:05.499366 28850 raft_consensus.cc:374] T 2ddf397fabe8453b8e1601a863a1b84e P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:05.501101 28849 tablet_bootstrap.cc:492] T 21dd29f1917a47f7ab496136a4c7ca07 P a068ef6770f14874bd1acb01bce2c329: Bootstrap starting.
I20240501 20:00:05.501168 28846 tablet_bootstrap.cc:654] T 6a175ef3d4c94d459208ea5cfc31e9d8 P dabdbfb59e8645c297a1696630198b6e: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:05.499713 28850 raft_consensus.cc:729] T 2ddf397fabe8453b8e1601a863a1b84e P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d7668f070e74286b38fda9d898f1acb, State: Initialized, Role: FOLLOWER
I20240501 20:00:05.502892 28850 consensus_queue.cc:260] T 2ddf397fabe8453b8e1601a863a1b84e P 3d7668f070e74286b38fda9d898f1acb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } }
I20240501 20:00:05.505246 28850 ts_tablet_manager.cc:1398] T 2ddf397fabe8453b8e1601a863a1b84e P 3d7668f070e74286b38fda9d898f1acb: Time spent starting tablet: real 0.020s user 0.005s sys 0.000s
I20240501 20:00:05.506121 28850 tablet_bootstrap.cc:492] T 101b282620ad41b28af298749f60ec52 P 3d7668f070e74286b38fda9d898f1acb: Bootstrap starting.
I20240501 20:00:05.506284 28858 raft_consensus.cc:482] T 8a0820e7c49f40f89038e0af73567bac P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:05.506718 28858 raft_consensus.cc:504] T 8a0820e7c49f40f89038e0af73567bac P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.509829 28728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0820e7c49f40f89038e0af73567bac" candidate_uuid: "3d7668f070e74286b38fda9d898f1acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a068ef6770f14874bd1acb01bce2c329"
I20240501 20:00:05.510900 28729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0820e7c49f40f89038e0af73567bac" candidate_uuid: "3d7668f070e74286b38fda9d898f1acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a068ef6770f14874bd1acb01bce2c329" is_pre_election: true
I20240501 20:00:05.512212 28728 raft_consensus.cc:3037] T 8a0820e7c49f40f89038e0af73567bac P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:05.509109 28549 catalog_manager.cc:5484] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e reported cstate change: term changed from 0 to 1, leader changed from <none> to dabdbfb59e8645c297a1696630198b6e (127.27.217.193). New cstate: current_term: 1 leader_uuid: "dabdbfb59e8645c297a1696630198b6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } health_report { overall_health: HEALTHY } } }
I20240501 20:00:05.515372 28858 leader_election.cc:288] T 8a0820e7c49f40f89038e0af73567bac P a068ef6770f14874bd1acb01bce2c329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dabdbfb59e8645c297a1696630198b6e (127.27.217.193:46801), 3d7668f070e74286b38fda9d898f1acb (127.27.217.195:38723)
I20240501 20:00:05.514253 28653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0820e7c49f40f89038e0af73567bac" candidate_uuid: "a068ef6770f14874bd1acb01bce2c329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dabdbfb59e8645c297a1696630198b6e" is_pre_election: true
I20240501 20:00:05.520654 28803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0820e7c49f40f89038e0af73567bac" candidate_uuid: "a068ef6770f14874bd1acb01bce2c329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d7668f070e74286b38fda9d898f1acb" is_pre_election: true
I20240501 20:00:05.523687 28850 tablet_bootstrap.cc:654] T 101b282620ad41b28af298749f60ec52 P 3d7668f070e74286b38fda9d898f1acb: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:05.524303 28803 raft_consensus.cc:2379] T 8a0820e7c49f40f89038e0af73567bac P 3d7668f070e74286b38fda9d898f1acb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a068ef6770f14874bd1acb01bce2c329 in current term 1: Already voted for candidate 3d7668f070e74286b38fda9d898f1acb in this term.
I20240501 20:00:05.526681 28692 leader_election.cc:302] T 8a0820e7c49f40f89038e0af73567bac P a068ef6770f14874bd1acb01bce2c329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a068ef6770f14874bd1acb01bce2c329; no voters: 3d7668f070e74286b38fda9d898f1acb, dabdbfb59e8645c297a1696630198b6e
I20240501 20:00:05.539196 28846 tablet_bootstrap.cc:492] T 6a175ef3d4c94d459208ea5cfc31e9d8 P dabdbfb59e8645c297a1696630198b6e: No bootstrap required, opened a new log
I20240501 20:00:05.539664 28846 ts_tablet_manager.cc:1367] T 6a175ef3d4c94d459208ea5cfc31e9d8 P dabdbfb59e8645c297a1696630198b6e: Time spent bootstrapping tablet: real 0.054s user 0.010s sys 0.003s
I20240501 20:00:05.541355 28728 raft_consensus.cc:2454] T 8a0820e7c49f40f89038e0af73567bac P a068ef6770f14874bd1acb01bce2c329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d7668f070e74286b38fda9d898f1acb in term 1.
I20240501 20:00:05.541888 28654 raft_consensus.cc:2454] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d7668f070e74286b38fda9d898f1acb in term 1.
I20240501 20:00:05.542003 28858 raft_consensus.cc:2726] T 8a0820e7c49f40f89038e0af73567bac P a068ef6770f14874bd1acb01bce2c329 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:00:05.542932 28849 tablet_bootstrap.cc:654] T 21dd29f1917a47f7ab496136a4c7ca07 P a068ef6770f14874bd1acb01bce2c329: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:05.543260 28766 leader_election.cc:302] T 8a0820e7c49f40f89038e0af73567bac P 3d7668f070e74286b38fda9d898f1acb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d7668f070e74286b38fda9d898f1acb, a068ef6770f14874bd1acb01bce2c329; no voters:
I20240501 20:00:05.544018 28859 raft_consensus.cc:2781] T 8a0820e7c49f40f89038e0af73567bac P 3d7668f070e74286b38fda9d898f1acb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:00:05.547183 28846 raft_consensus.cc:348] T 6a175ef3d4c94d459208ea5cfc31e9d8 P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.548012 28846 raft_consensus.cc:374] T 6a175ef3d4c94d459208ea5cfc31e9d8 P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:05.548373 28846 raft_consensus.cc:729] T 6a175ef3d4c94d459208ea5cfc31e9d8 P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dabdbfb59e8645c297a1696630198b6e, State: Initialized, Role: FOLLOWER
I20240501 20:00:05.549154 28846 consensus_queue.cc:260] T 6a175ef3d4c94d459208ea5cfc31e9d8 P dabdbfb59e8645c297a1696630198b6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.551349 28846 ts_tablet_manager.cc:1398] T 6a175ef3d4c94d459208ea5cfc31e9d8 P dabdbfb59e8645c297a1696630198b6e: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20240501 20:00:05.552313 28846 tablet_bootstrap.cc:492] T 676f538543c2491399b66b7db1490445 P dabdbfb59e8645c297a1696630198b6e: Bootstrap starting.
I20240501 20:00:05.553165 28877 raft_consensus.cc:482] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:05.553594 28877 raft_consensus.cc:504] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.555034 28859 raft_consensus.cc:686] T 8a0820e7c49f40f89038e0af73567bac P 3d7668f070e74286b38fda9d898f1acb [term 1 LEADER]: Becoming Leader. State: Replica: 3d7668f070e74286b38fda9d898f1acb, State: Running, Role: LEADER
I20240501 20:00:05.556507 28654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21dd29f1917a47f7ab496136a4c7ca07" candidate_uuid: "3d7668f070e74286b38fda9d898f1acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dabdbfb59e8645c297a1696630198b6e" is_pre_election: true
I20240501 20:00:05.556317 28859 consensus_queue.cc:237] T 8a0820e7c49f40f89038e0af73567bac P 3d7668f070e74286b38fda9d898f1acb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.557145 28654 raft_consensus.cc:2454] T 21dd29f1917a47f7ab496136a4c7ca07 P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d7668f070e74286b38fda9d898f1acb in term 0.
I20240501 20:00:05.558290 28768 leader_election.cc:302] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d7668f070e74286b38fda9d898f1acb, dabdbfb59e8645c297a1696630198b6e; no voters:
I20240501 20:00:05.559355 28877 leader_election.cc:288] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dabdbfb59e8645c297a1696630198b6e (127.27.217.193:46801), a068ef6770f14874bd1acb01bce2c329 (127.27.217.194:37307)
I20240501 20:00:05.560930 28728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21dd29f1917a47f7ab496136a4c7ca07" candidate_uuid: "3d7668f070e74286b38fda9d898f1acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a068ef6770f14874bd1acb01bce2c329" is_pre_election: true
W20240501 20:00:05.562197 28766 leader_election.cc:341] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a068ef6770f14874bd1acb01bce2c329 (127.27.217.194:37307): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:00:05.560570 28877 raft_consensus.cc:2781] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:00:05.563459 28877 raft_consensus.cc:482] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:00:05.563856 28877 raft_consensus.cc:3037] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:05.564450 28850 tablet_bootstrap.cc:492] T 101b282620ad41b28af298749f60ec52 P 3d7668f070e74286b38fda9d898f1acb: No bootstrap required, opened a new log
I20240501 20:00:05.564471 28846 tablet_bootstrap.cc:654] T 676f538543c2491399b66b7db1490445 P dabdbfb59e8645c297a1696630198b6e: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:05.568843 28850 ts_tablet_manager.cc:1367] T 101b282620ad41b28af298749f60ec52 P 3d7668f070e74286b38fda9d898f1acb: Time spent bootstrapping tablet: real 0.063s user 0.012s sys 0.002s
I20240501 20:00:05.579391 28858 raft_consensus.cc:482] T 14bfbbeb889d43218d5807488de4fed2 P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:05.579846 28858 raft_consensus.cc:504] T 14bfbbeb889d43218d5807488de4fed2 P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.580758 28849 tablet_bootstrap.cc:492] T 21dd29f1917a47f7ab496136a4c7ca07 P a068ef6770f14874bd1acb01bce2c329: No bootstrap required, opened a new log
I20240501 20:00:05.581297 28849 ts_tablet_manager.cc:1367] T 21dd29f1917a47f7ab496136a4c7ca07 P a068ef6770f14874bd1acb01bce2c329: Time spent bootstrapping tablet: real 0.080s user 0.012s sys 0.001s
I20240501 20:00:05.581820 28858 leader_election.cc:288] T 14bfbbeb889d43218d5807488de4fed2 P a068ef6770f14874bd1acb01bce2c329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dabdbfb59e8645c297a1696630198b6e (127.27.217.193:46801), 3d7668f070e74286b38fda9d898f1acb (127.27.217.195:38723)
I20240501 20:00:05.588253 28654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14bfbbeb889d43218d5807488de4fed2" candidate_uuid: "a068ef6770f14874bd1acb01bce2c329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dabdbfb59e8645c297a1696630198b6e" is_pre_election: true
I20240501 20:00:05.588253 28803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14bfbbeb889d43218d5807488de4fed2" candidate_uuid: "a068ef6770f14874bd1acb01bce2c329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d7668f070e74286b38fda9d898f1acb" is_pre_election: true
I20240501 20:00:05.589385 28803 raft_consensus.cc:2454] T 14bfbbeb889d43218d5807488de4fed2 P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a068ef6770f14874bd1acb01bce2c329 in term 0.
W20240501 20:00:05.590214 28694 leader_election.cc:341] T 14bfbbeb889d43218d5807488de4fed2 P a068ef6770f14874bd1acb01bce2c329 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dabdbfb59e8645c297a1696630198b6e (127.27.217.193:46801): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:00:05.590785 28692 leader_election.cc:302] T 14bfbbeb889d43218d5807488de4fed2 P a068ef6770f14874bd1acb01bce2c329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d7668f070e74286b38fda9d898f1acb, a068ef6770f14874bd1acb01bce2c329; no voters: dabdbfb59e8645c297a1696630198b6e
I20240501 20:00:05.591370 28858 raft_consensus.cc:2781] T 14bfbbeb889d43218d5807488de4fed2 P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:00:05.590847 28850 raft_consensus.cc:348] T 101b282620ad41b28af298749f60ec52 P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.591886 28858 raft_consensus.cc:482] T 14bfbbeb889d43218d5807488de4fed2 P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:00:05.592271 28850 raft_consensus.cc:374] T 101b282620ad41b28af298749f60ec52 P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:05.592451 28858 raft_consensus.cc:3037] T 14bfbbeb889d43218d5807488de4fed2 P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:05.592635 28850 raft_consensus.cc:729] T 101b282620ad41b28af298749f60ec52 P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d7668f070e74286b38fda9d898f1acb, State: Initialized, Role: FOLLOWER
I20240501 20:00:05.593426 28850 consensus_queue.cc:260] T 101b282620ad41b28af298749f60ec52 P 3d7668f070e74286b38fda9d898f1acb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.595261 28850 ts_tablet_manager.cc:1398] T 101b282620ad41b28af298749f60ec52 P 3d7668f070e74286b38fda9d898f1acb: Time spent starting tablet: real 0.025s user 0.003s sys 0.001s
I20240501 20:00:05.595922 28850 tablet_bootstrap.cc:492] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb: Bootstrap starting.
I20240501 20:00:05.597518 28858 raft_consensus.cc:504] T 14bfbbeb889d43218d5807488de4fed2 P a068ef6770f14874bd1acb01bce2c329 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.599402 28877 raft_consensus.cc:504] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.600014 28654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14bfbbeb889d43218d5807488de4fed2" candidate_uuid: "a068ef6770f14874bd1acb01bce2c329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dabdbfb59e8645c297a1696630198b6e"
I20240501 20:00:05.601533 28850 tablet_bootstrap.cc:654] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:05.601650 28877 leader_election.cc:288] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb [CANDIDATE]: Term 1 election: Requested vote from peers dabdbfb59e8645c297a1696630198b6e (127.27.217.193:46801), a068ef6770f14874bd1acb01bce2c329 (127.27.217.194:37307)
I20240501 20:00:05.603281 28849 raft_consensus.cc:348] T 21dd29f1917a47f7ab496136a4c7ca07 P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.603986 28849 raft_consensus.cc:374] T 21dd29f1917a47f7ab496136a4c7ca07 P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:05.604285 28849 raft_consensus.cc:729] T 21dd29f1917a47f7ab496136a4c7ca07 P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a068ef6770f14874bd1acb01bce2c329, State: Initialized, Role: FOLLOWER
I20240501 20:00:05.604990 28849 consensus_queue.cc:260] T 21dd29f1917a47f7ab496136a4c7ca07 P a068ef6770f14874bd1acb01bce2c329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.608367 28728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21dd29f1917a47f7ab496136a4c7ca07" candidate_uuid: "3d7668f070e74286b38fda9d898f1acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a068ef6770f14874bd1acb01bce2c329"
I20240501 20:00:05.608704 28846 tablet_bootstrap.cc:492] T 676f538543c2491399b66b7db1490445 P dabdbfb59e8645c297a1696630198b6e: No bootstrap required, opened a new log
I20240501 20:00:05.608901 28728 raft_consensus.cc:3037] T 21dd29f1917a47f7ab496136a4c7ca07 P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:05.609149 28846 ts_tablet_manager.cc:1367] T 676f538543c2491399b66b7db1490445 P dabdbfb59e8645c297a1696630198b6e: Time spent bootstrapping tablet: real 0.057s user 0.016s sys 0.006s
I20240501 20:00:05.610174 28849 ts_tablet_manager.cc:1398] T 21dd29f1917a47f7ab496136a4c7ca07 P a068ef6770f14874bd1acb01bce2c329: Time spent starting tablet: real 0.028s user 0.007s sys 0.000s
I20240501 20:00:05.613131 28846 raft_consensus.cc:348] T 676f538543c2491399b66b7db1490445 P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.614073 28846 raft_consensus.cc:374] T 676f538543c2491399b66b7db1490445 P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:05.614436 28846 raft_consensus.cc:729] T 676f538543c2491399b66b7db1490445 P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dabdbfb59e8645c297a1696630198b6e, State: Initialized, Role: FOLLOWER
I20240501 20:00:05.615322 28728 raft_consensus.cc:2454] T 21dd29f1917a47f7ab496136a4c7ca07 P a068ef6770f14874bd1acb01bce2c329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d7668f070e74286b38fda9d898f1acb in term 1.
W20240501 20:00:05.601346 28694 leader_election.cc:341] T 14bfbbeb889d43218d5807488de4fed2 P a068ef6770f14874bd1acb01bce2c329 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer dabdbfb59e8645c297a1696630198b6e (127.27.217.193:46801): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:00:05.622095 28550 catalog_manager.cc:5484] T 8a0820e7c49f40f89038e0af73567bac P 3d7668f070e74286b38fda9d898f1acb reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d7668f070e74286b38fda9d898f1acb (127.27.217.195). New cstate: current_term: 1 leader_uuid: "3d7668f070e74286b38fda9d898f1acb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } health_report { overall_health: HEALTHY } } }
I20240501 20:00:05.616379 28849 tablet_bootstrap.cc:492] T fbcced7b903441278c70c4d3a1458be4 P a068ef6770f14874bd1acb01bce2c329: Bootstrap starting.
I20240501 20:00:05.616621 28766 leader_election.cc:302] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d7668f070e74286b38fda9d898f1acb, a068ef6770f14874bd1acb01bce2c329; no voters:
I20240501 20:00:05.623656 28654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21dd29f1917a47f7ab496136a4c7ca07" candidate_uuid: "3d7668f070e74286b38fda9d898f1acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dabdbfb59e8645c297a1696630198b6e"
I20240501 20:00:05.627048 28877 raft_consensus.cc:2781] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:00:05.627314 28654 raft_consensus.cc:3037] T 21dd29f1917a47f7ab496136a4c7ca07 P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:05.627687 28877 raft_consensus.cc:686] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb [term 1 LEADER]: Becoming Leader. State: Replica: 3d7668f070e74286b38fda9d898f1acb, State: Running, Role: LEADER
I20240501 20:00:05.615249 28846 consensus_queue.cc:260] T 676f538543c2491399b66b7db1490445 P dabdbfb59e8645c297a1696630198b6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.628439 28877 consensus_queue.cc:237] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.629673 28846 ts_tablet_manager.cc:1398] T 676f538543c2491399b66b7db1490445 P dabdbfb59e8645c297a1696630198b6e: Time spent starting tablet: real 0.020s user 0.005s sys 0.003s
I20240501 20:00:05.630514 28846 tablet_bootstrap.cc:492] T fbcced7b903441278c70c4d3a1458be4 P dabdbfb59e8645c297a1696630198b6e: Bootstrap starting.
I20240501 20:00:05.634516 28654 raft_consensus.cc:2454] T 21dd29f1917a47f7ab496136a4c7ca07 P dabdbfb59e8645c297a1696630198b6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d7668f070e74286b38fda9d898f1acb in term 1.
I20240501 20:00:05.635969 28846 tablet_bootstrap.cc:654] T fbcced7b903441278c70c4d3a1458be4 P dabdbfb59e8645c297a1696630198b6e: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:05.644032 28858 leader_election.cc:288] T 14bfbbeb889d43218d5807488de4fed2 P a068ef6770f14874bd1acb01bce2c329 [CANDIDATE]: Term 1 election: Requested vote from peers dabdbfb59e8645c297a1696630198b6e (127.27.217.193:46801), 3d7668f070e74286b38fda9d898f1acb (127.27.217.195:38723)
I20240501 20:00:05.645897 28803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14bfbbeb889d43218d5807488de4fed2" candidate_uuid: "a068ef6770f14874bd1acb01bce2c329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d7668f070e74286b38fda9d898f1acb"
I20240501 20:00:05.646493 28803 raft_consensus.cc:3037] T 14bfbbeb889d43218d5807488de4fed2 P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:05.647536 28863 raft_consensus.cc:482] T 8bd1afda996d441b8d3eaa64dd36d1c7 P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:05.648213 28863 raft_consensus.cc:504] T 8bd1afda996d441b8d3eaa64dd36d1c7 P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.651520 28863 leader_election.cc:288] T 8bd1afda996d441b8d3eaa64dd36d1c7 P dabdbfb59e8645c297a1696630198b6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a068ef6770f14874bd1acb01bce2c329 (127.27.217.194:37307), 3d7668f070e74286b38fda9d898f1acb (127.27.217.195:38723)
I20240501 20:00:05.653510 28802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd1afda996d441b8d3eaa64dd36d1c7" candidate_uuid: "dabdbfb59e8645c297a1696630198b6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d7668f070e74286b38fda9d898f1acb" is_pre_election: true
I20240501 20:00:05.657506 28728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd1afda996d441b8d3eaa64dd36d1c7" candidate_uuid: "dabdbfb59e8645c297a1696630198b6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a068ef6770f14874bd1acb01bce2c329" is_pre_election: true
W20240501 20:00:05.657711 28617 leader_election.cc:341] T 8bd1afda996d441b8d3eaa64dd36d1c7 P dabdbfb59e8645c297a1696630198b6e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d7668f070e74286b38fda9d898f1acb (127.27.217.195:38723): Illegal state: must be running to vote when last-logged opid is not known
W20240501 20:00:05.659206 28617 leader_election.cc:341] T 8bd1afda996d441b8d3eaa64dd36d1c7 P dabdbfb59e8645c297a1696630198b6e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a068ef6770f14874bd1acb01bce2c329 (127.27.217.194:37307): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:00:05.659667 28617 leader_election.cc:302] T 8bd1afda996d441b8d3eaa64dd36d1c7 P dabdbfb59e8645c297a1696630198b6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dabdbfb59e8645c297a1696630198b6e; no voters: 3d7668f070e74286b38fda9d898f1acb, a068ef6770f14874bd1acb01bce2c329
I20240501 20:00:05.660655 28863 raft_consensus.cc:2726] T 8bd1afda996d441b8d3eaa64dd36d1c7 P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:00:05.663774 28846 tablet_bootstrap.cc:492] T fbcced7b903441278c70c4d3a1458be4 P dabdbfb59e8645c297a1696630198b6e: No bootstrap required, opened a new log
I20240501 20:00:05.664232 28846 ts_tablet_manager.cc:1367] T fbcced7b903441278c70c4d3a1458be4 P dabdbfb59e8645c297a1696630198b6e: Time spent bootstrapping tablet: real 0.034s user 0.016s sys 0.000s
I20240501 20:00:05.664796 28803 raft_consensus.cc:2454] T 14bfbbeb889d43218d5807488de4fed2 P 3d7668f070e74286b38fda9d898f1acb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a068ef6770f14874bd1acb01bce2c329 in term 1.
I20240501 20:00:05.666725 28846 raft_consensus.cc:348] T fbcced7b903441278c70c4d3a1458be4 P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.665990 28692 leader_election.cc:302] T 14bfbbeb889d43218d5807488de4fed2 P a068ef6770f14874bd1acb01bce2c329 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d7668f070e74286b38fda9d898f1acb, a068ef6770f14874bd1acb01bce2c329; no voters: dabdbfb59e8645c297a1696630198b6e
I20240501 20:00:05.668196 28858 raft_consensus.cc:2781] T 14bfbbeb889d43218d5807488de4fed2 P a068ef6770f14874bd1acb01bce2c329 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:00:05.668637 28858 raft_consensus.cc:686] T 14bfbbeb889d43218d5807488de4fed2 P a068ef6770f14874bd1acb01bce2c329 [term 1 LEADER]: Becoming Leader. State: Replica: a068ef6770f14874bd1acb01bce2c329, State: Running, Role: LEADER
I20240501 20:00:05.668943 28849 tablet_bootstrap.cc:654] T fbcced7b903441278c70c4d3a1458be4 P a068ef6770f14874bd1acb01bce2c329: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:05.669343 28858 consensus_queue.cc:237] T 14bfbbeb889d43218d5807488de4fed2 P a068ef6770f14874bd1acb01bce2c329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.669965 28550 catalog_manager.cc:5484] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d7668f070e74286b38fda9d898f1acb (127.27.217.195). New cstate: current_term: 1 leader_uuid: "3d7668f070e74286b38fda9d898f1acb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } health_report { overall_health: HEALTHY } } }
I20240501 20:00:05.667414 28846 raft_consensus.cc:374] T fbcced7b903441278c70c4d3a1458be4 P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:05.672870 28846 raft_consensus.cc:729] T fbcced7b903441278c70c4d3a1458be4 P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dabdbfb59e8645c297a1696630198b6e, State: Initialized, Role: FOLLOWER
I20240501 20:00:05.675251 28850 tablet_bootstrap.cc:492] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb: No bootstrap required, opened a new log
I20240501 20:00:05.675179 28846 consensus_queue.cc:260] T fbcced7b903441278c70c4d3a1458be4 P dabdbfb59e8645c297a1696630198b6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.675755 28850 ts_tablet_manager.cc:1367] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb: Time spent bootstrapping tablet: real 0.080s user 0.012s sys 0.030s
I20240501 20:00:05.677238 28846 ts_tablet_manager.cc:1398] T fbcced7b903441278c70c4d3a1458be4 P dabdbfb59e8645c297a1696630198b6e: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20240501 20:00:05.678222 28846 tablet_bootstrap.cc:492] T 14bfbbeb889d43218d5807488de4fed2 P dabdbfb59e8645c297a1696630198b6e: Bootstrap starting.
I20240501 20:00:05.682878 28846 tablet_bootstrap.cc:654] T 14bfbbeb889d43218d5807488de4fed2 P dabdbfb59e8645c297a1696630198b6e: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:05.686604 28849 tablet_bootstrap.cc:492] T fbcced7b903441278c70c4d3a1458be4 P a068ef6770f14874bd1acb01bce2c329: No bootstrap required, opened a new log
I20240501 20:00:05.687088 28849 ts_tablet_manager.cc:1367] T fbcced7b903441278c70c4d3a1458be4 P a068ef6770f14874bd1acb01bce2c329: Time spent bootstrapping tablet: real 0.071s user 0.007s sys 0.003s
I20240501 20:00:05.686928 28551 catalog_manager.cc:5484] T 14bfbbeb889d43218d5807488de4fed2 P a068ef6770f14874bd1acb01bce2c329 reported cstate change: term changed from 0 to 1, leader changed from <none> to a068ef6770f14874bd1acb01bce2c329 (127.27.217.194). New cstate: current_term: 1 leader_uuid: "a068ef6770f14874bd1acb01bce2c329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } health_report { overall_health: UNKNOWN } } }
I20240501 20:00:05.689711 28849 raft_consensus.cc:348] T fbcced7b903441278c70c4d3a1458be4 P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.690250 28849 raft_consensus.cc:374] T fbcced7b903441278c70c4d3a1458be4 P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:05.690474 28849 raft_consensus.cc:729] T fbcced7b903441278c70c4d3a1458be4 P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a068ef6770f14874bd1acb01bce2c329, State: Initialized, Role: FOLLOWER
I20240501 20:00:05.691107 28849 consensus_queue.cc:260] T fbcced7b903441278c70c4d3a1458be4 P a068ef6770f14874bd1acb01bce2c329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.693367 28849 ts_tablet_manager.cc:1398] T fbcced7b903441278c70c4d3a1458be4 P a068ef6770f14874bd1acb01bce2c329: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20240501 20:00:05.692817 28850 raft_consensus.cc:348] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.693715 28850 raft_consensus.cc:374] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:05.694000 28850 raft_consensus.cc:729] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d7668f070e74286b38fda9d898f1acb, State: Initialized, Role: FOLLOWER
I20240501 20:00:05.694185 28849 tablet_bootstrap.cc:492] T 676f538543c2491399b66b7db1490445 P a068ef6770f14874bd1acb01bce2c329: Bootstrap starting.
I20240501 20:00:05.694646 28850 consensus_queue.cc:260] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.698506 28850 ts_tablet_manager.cc:1398] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb: Time spent starting tablet: real 0.022s user 0.004s sys 0.001s
I20240501 20:00:05.699317 28850 tablet_bootstrap.cc:492] T 8bd1afda996d441b8d3eaa64dd36d1c7 P 3d7668f070e74286b38fda9d898f1acb: Bootstrap starting.
I20240501 20:00:05.702134 28877 raft_consensus.cc:482] T 2ddf397fabe8453b8e1601a863a1b84e P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:05.702551 28877 raft_consensus.cc:504] T 2ddf397fabe8453b8e1601a863a1b84e P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } }
I20240501 20:00:05.704296 28849 tablet_bootstrap.cc:654] T 676f538543c2491399b66b7db1490445 P a068ef6770f14874bd1acb01bce2c329: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:05.704238 28877 leader_election.cc:288] T 2ddf397fabe8453b8e1601a863a1b84e P 3d7668f070e74286b38fda9d898f1acb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a068ef6770f14874bd1acb01bce2c329 (127.27.217.194:37307), dabdbfb59e8645c297a1696630198b6e (127.27.217.193:46801)
I20240501 20:00:05.704973 28863 raft_consensus.cc:482] T 101b282620ad41b28af298749f60ec52 P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:05.705519 28728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ddf397fabe8453b8e1601a863a1b84e" candidate_uuid: "3d7668f070e74286b38fda9d898f1acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a068ef6770f14874bd1acb01bce2c329" is_pre_election: true
I20240501 20:00:05.705394 28863 raft_consensus.cc:504] T 101b282620ad41b28af298749f60ec52 P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.706215 28728 raft_consensus.cc:2379] T 2ddf397fabe8453b8e1601a863a1b84e P a068ef6770f14874bd1acb01bce2c329 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d7668f070e74286b38fda9d898f1acb in current term 1: Already voted for candidate dabdbfb59e8645c297a1696630198b6e in this term.
I20240501 20:00:05.707188 28863 leader_election.cc:288] T 101b282620ad41b28af298749f60ec52 P dabdbfb59e8645c297a1696630198b6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a068ef6770f14874bd1acb01bce2c329 (127.27.217.194:37307), 3d7668f070e74286b38fda9d898f1acb (127.27.217.195:38723)
I20240501 20:00:05.707819 28654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ddf397fabe8453b8e1601a863a1b84e" candidate_uuid: "3d7668f070e74286b38fda9d898f1acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dabdbfb59e8645c297a1696630198b6e" is_pre_election: true
I20240501 20:00:05.708143 28803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "101b282620ad41b28af298749f60ec52" candidate_uuid: "dabdbfb59e8645c297a1696630198b6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d7668f070e74286b38fda9d898f1acb" is_pre_election: true
I20240501 20:00:05.708673 28803 raft_consensus.cc:2454] T 101b282620ad41b28af298749f60ec52 P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dabdbfb59e8645c297a1696630198b6e in term 0.
I20240501 20:00:05.707832 28728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "101b282620ad41b28af298749f60ec52" candidate_uuid: "dabdbfb59e8645c297a1696630198b6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a068ef6770f14874bd1acb01bce2c329" is_pre_election: true
I20240501 20:00:05.709852 28728 raft_consensus.cc:2454] T 101b282620ad41b28af298749f60ec52 P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dabdbfb59e8645c297a1696630198b6e in term 0.
I20240501 20:00:05.710140 28617 leader_election.cc:302] T 101b282620ad41b28af298749f60ec52 P dabdbfb59e8645c297a1696630198b6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d7668f070e74286b38fda9d898f1acb, dabdbfb59e8645c297a1696630198b6e; no voters:
I20240501 20:00:05.709874 28768 leader_election.cc:302] T 2ddf397fabe8453b8e1601a863a1b84e P 3d7668f070e74286b38fda9d898f1acb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d7668f070e74286b38fda9d898f1acb; no voters: a068ef6770f14874bd1acb01bce2c329, dabdbfb59e8645c297a1696630198b6e
I20240501 20:00:05.711053 28863 raft_consensus.cc:2781] T 101b282620ad41b28af298749f60ec52 P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:00:05.711215 28877 raft_consensus.cc:3037] T 2ddf397fabe8453b8e1601a863a1b84e P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:05.711406 28863 raft_consensus.cc:482] T 101b282620ad41b28af298749f60ec52 P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:00:05.711812 28863 raft_consensus.cc:3037] T 101b282620ad41b28af298749f60ec52 P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:05.718828 28877 raft_consensus.cc:2726] T 2ddf397fabe8453b8e1601a863a1b84e P 3d7668f070e74286b38fda9d898f1acb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:00:05.721542 28846 tablet_bootstrap.cc:492] T 14bfbbeb889d43218d5807488de4fed2 P dabdbfb59e8645c297a1696630198b6e: No bootstrap required, opened a new log
I20240501 20:00:05.718815 28863 raft_consensus.cc:504] T 101b282620ad41b28af298749f60ec52 P dabdbfb59e8645c297a1696630198b6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.721910 28846 ts_tablet_manager.cc:1367] T 14bfbbeb889d43218d5807488de4fed2 P dabdbfb59e8645c297a1696630198b6e: Time spent bootstrapping tablet: real 0.044s user 0.011s sys 0.004s
I20240501 20:00:05.723317 28863 leader_election.cc:288] T 101b282620ad41b28af298749f60ec52 P dabdbfb59e8645c297a1696630198b6e [CANDIDATE]: Term 1 election: Requested vote from peers a068ef6770f14874bd1acb01bce2c329 (127.27.217.194:37307), 3d7668f070e74286b38fda9d898f1acb (127.27.217.195:38723)
I20240501 20:00:05.724290 28803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "101b282620ad41b28af298749f60ec52" candidate_uuid: "dabdbfb59e8645c297a1696630198b6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d7668f070e74286b38fda9d898f1acb"
I20240501 20:00:05.729460 28858 raft_consensus.cc:482] T fbcced7b903441278c70c4d3a1458be4 P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:05.725313 28728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "101b282620ad41b28af298749f60ec52" candidate_uuid: "dabdbfb59e8645c297a1696630198b6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a068ef6770f14874bd1acb01bce2c329"
I20240501 20:00:05.730130 28728 raft_consensus.cc:3037] T 101b282620ad41b28af298749f60ec52 P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:05.729949 28858 raft_consensus.cc:504] T fbcced7b903441278c70c4d3a1458be4 P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.730643 28849 tablet_bootstrap.cc:492] T 676f538543c2491399b66b7db1490445 P a068ef6770f14874bd1acb01bce2c329: No bootstrap required, opened a new log
I20240501 20:00:05.731082 28849 ts_tablet_manager.cc:1367] T 676f538543c2491399b66b7db1490445 P a068ef6770f14874bd1acb01bce2c329: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.014s
I20240501 20:00:05.731806 28858 leader_election.cc:288] T fbcced7b903441278c70c4d3a1458be4 P a068ef6770f14874bd1acb01bce2c329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dabdbfb59e8645c297a1696630198b6e (127.27.217.193:46801), 3d7668f070e74286b38fda9d898f1acb (127.27.217.195:38723)
I20240501 20:00:05.733203 28654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbcced7b903441278c70c4d3a1458be4" candidate_uuid: "a068ef6770f14874bd1acb01bce2c329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dabdbfb59e8645c297a1696630198b6e" is_pre_election: true
I20240501 20:00:05.733423 28849 raft_consensus.cc:348] T 676f538543c2491399b66b7db1490445 P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.733779 28654 raft_consensus.cc:2454] T fbcced7b903441278c70c4d3a1458be4 P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a068ef6770f14874bd1acb01bce2c329 in term 0.
I20240501 20:00:05.734086 28849 raft_consensus.cc:374] T 676f538543c2491399b66b7db1490445 P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:05.734391 28849 raft_consensus.cc:729] T 676f538543c2491399b66b7db1490445 P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a068ef6770f14874bd1acb01bce2c329, State: Initialized, Role: FOLLOWER
I20240501 20:00:05.728698 28850 tablet_bootstrap.cc:654] T 8bd1afda996d441b8d3eaa64dd36d1c7 P 3d7668f070e74286b38fda9d898f1acb: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:05.729547 28803 raft_consensus.cc:3037] T 101b282620ad41b28af298749f60ec52 P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:05.735319 28802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbcced7b903441278c70c4d3a1458be4" candidate_uuid: "a068ef6770f14874bd1acb01bce2c329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d7668f070e74286b38fda9d898f1acb" is_pre_election: true
I20240501 20:00:05.736586 28694 leader_election.cc:302] T fbcced7b903441278c70c4d3a1458be4 P a068ef6770f14874bd1acb01bce2c329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a068ef6770f14874bd1acb01bce2c329, dabdbfb59e8645c297a1696630198b6e; no voters:
W20240501 20:00:05.737696 28692 leader_election.cc:341] T fbcced7b903441278c70c4d3a1458be4 P a068ef6770f14874bd1acb01bce2c329 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d7668f070e74286b38fda9d898f1acb (127.27.217.195:38723): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:00:05.738536 28858 raft_consensus.cc:2781] T fbcced7b903441278c70c4d3a1458be4 P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:00:05.738257 28849 consensus_queue.cc:260] T 676f538543c2491399b66b7db1490445 P a068ef6770f14874bd1acb01bce2c329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.738868 28858 raft_consensus.cc:482] T fbcced7b903441278c70c4d3a1458be4 P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:00:05.739212 28858 raft_consensus.cc:3037] T fbcced7b903441278c70c4d3a1458be4 P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:05.740950 28849 ts_tablet_manager.cc:1398] T 676f538543c2491399b66b7db1490445 P a068ef6770f14874bd1acb01bce2c329: Time spent starting tablet: real 0.010s user 0.004s sys 0.002s
I20240501 20:00:05.724753 28846 raft_consensus.cc:348] T 14bfbbeb889d43218d5807488de4fed2 P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.741833 28849 tablet_bootstrap.cc:492] T 6a175ef3d4c94d459208ea5cfc31e9d8 P a068ef6770f14874bd1acb01bce2c329: Bootstrap starting.
I20240501 20:00:05.741533 28846 raft_consensus.cc:374] T 14bfbbeb889d43218d5807488de4fed2 P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:05.743000 28846 raft_consensus.cc:729] T 14bfbbeb889d43218d5807488de4fed2 P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dabdbfb59e8645c297a1696630198b6e, State: Initialized, Role: FOLLOWER
I20240501 20:00:05.744540 28803 raft_consensus.cc:2454] T 101b282620ad41b28af298749f60ec52 P 3d7668f070e74286b38fda9d898f1acb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dabdbfb59e8645c297a1696630198b6e in term 1.
I20240501 20:00:05.745609 28617 leader_election.cc:302] T 101b282620ad41b28af298749f60ec52 P dabdbfb59e8645c297a1696630198b6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d7668f070e74286b38fda9d898f1acb, dabdbfb59e8645c297a1696630198b6e; no voters:
I20240501 20:00:05.746361 28728 raft_consensus.cc:2454] T 101b282620ad41b28af298749f60ec52 P a068ef6770f14874bd1acb01bce2c329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dabdbfb59e8645c297a1696630198b6e in term 1.
I20240501 20:00:05.759307 28858 raft_consensus.cc:504] T fbcced7b903441278c70c4d3a1458be4 P a068ef6770f14874bd1acb01bce2c329 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.761178 28863 raft_consensus.cc:2781] T 101b282620ad41b28af298749f60ec52 P dabdbfb59e8645c297a1696630198b6e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:00:05.761808 28863 raft_consensus.cc:686] T 101b282620ad41b28af298749f60ec52 P dabdbfb59e8645c297a1696630198b6e [term 1 LEADER]: Becoming Leader. State: Replica: dabdbfb59e8645c297a1696630198b6e, State: Running, Role: LEADER
I20240501 20:00:05.780544 28803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbcced7b903441278c70c4d3a1458be4" candidate_uuid: "a068ef6770f14874bd1acb01bce2c329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d7668f070e74286b38fda9d898f1acb"
W20240501 20:00:05.782543 28692 leader_election.cc:341] T fbcced7b903441278c70c4d3a1458be4 P a068ef6770f14874bd1acb01bce2c329 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3d7668f070e74286b38fda9d898f1acb (127.27.217.195:38723): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:00:05.783314 28654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbcced7b903441278c70c4d3a1458be4" candidate_uuid: "a068ef6770f14874bd1acb01bce2c329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dabdbfb59e8645c297a1696630198b6e"
I20240501 20:00:05.783877 28850 tablet_bootstrap.cc:492] T 8bd1afda996d441b8d3eaa64dd36d1c7 P 3d7668f070e74286b38fda9d898f1acb: No bootstrap required, opened a new log
I20240501 20:00:05.783891 28654 raft_consensus.cc:3037] T fbcced7b903441278c70c4d3a1458be4 P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:05.784039 28846 consensus_queue.cc:260] T 14bfbbeb889d43218d5807488de4fed2 P dabdbfb59e8645c297a1696630198b6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.784875 28850 ts_tablet_manager.cc:1367] T 8bd1afda996d441b8d3eaa64dd36d1c7 P 3d7668f070e74286b38fda9d898f1acb: Time spent bootstrapping tablet: real 0.086s user 0.011s sys 0.000s
I20240501 20:00:05.786437 28846 ts_tablet_manager.cc:1398] T 14bfbbeb889d43218d5807488de4fed2 P dabdbfb59e8645c297a1696630198b6e: Time spent starting tablet: real 0.064s user 0.010s sys 0.016s
I20240501 20:00:05.783957 28858 leader_election.cc:288] T fbcced7b903441278c70c4d3a1458be4 P a068ef6770f14874bd1acb01bce2c329 [CANDIDATE]: Term 1 election: Requested vote from peers dabdbfb59e8645c297a1696630198b6e (127.27.217.193:46801), 3d7668f070e74286b38fda9d898f1acb (127.27.217.195:38723)
I20240501 20:00:05.787938 28863 consensus_queue.cc:237] T 101b282620ad41b28af298749f60ec52 P dabdbfb59e8645c297a1696630198b6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.788110 28850 raft_consensus.cc:348] T 8bd1afda996d441b8d3eaa64dd36d1c7 P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.788920 28850 raft_consensus.cc:374] T 8bd1afda996d441b8d3eaa64dd36d1c7 P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:05.789232 28850 raft_consensus.cc:729] T 8bd1afda996d441b8d3eaa64dd36d1c7 P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d7668f070e74286b38fda9d898f1acb, State: Initialized, Role: FOLLOWER
I20240501 20:00:05.789898 28850 consensus_queue.cc:260] T 8bd1afda996d441b8d3eaa64dd36d1c7 P 3d7668f070e74286b38fda9d898f1acb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.790460 28654 raft_consensus.cc:2454] T fbcced7b903441278c70c4d3a1458be4 P dabdbfb59e8645c297a1696630198b6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a068ef6770f14874bd1acb01bce2c329 in term 1.
I20240501 20:00:05.791707 28694 leader_election.cc:302] T fbcced7b903441278c70c4d3a1458be4 P a068ef6770f14874bd1acb01bce2c329 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a068ef6770f14874bd1acb01bce2c329, dabdbfb59e8645c297a1696630198b6e; no voters: 3d7668f070e74286b38fda9d898f1acb
I20240501 20:00:05.792778 28858 raft_consensus.cc:2781] T fbcced7b903441278c70c4d3a1458be4 P a068ef6770f14874bd1acb01bce2c329 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:00:05.794618 28850 ts_tablet_manager.cc:1398] T 8bd1afda996d441b8d3eaa64dd36d1c7 P 3d7668f070e74286b38fda9d898f1acb: Time spent starting tablet: real 0.009s user 0.004s sys 0.002s
I20240501 20:00:05.795356 28849 tablet_bootstrap.cc:654] T 6a175ef3d4c94d459208ea5cfc31e9d8 P a068ef6770f14874bd1acb01bce2c329: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:05.795473 28850 tablet_bootstrap.cc:492] T fbcced7b903441278c70c4d3a1458be4 P 3d7668f070e74286b38fda9d898f1acb: Bootstrap starting.
I20240501 20:00:05.803790 28850 tablet_bootstrap.cc:654] T fbcced7b903441278c70c4d3a1458be4 P 3d7668f070e74286b38fda9d898f1acb: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:05.804358 28858 raft_consensus.cc:686] T fbcced7b903441278c70c4d3a1458be4 P a068ef6770f14874bd1acb01bce2c329 [term 1 LEADER]: Becoming Leader. State: Replica: a068ef6770f14874bd1acb01bce2c329, State: Running, Role: LEADER
I20240501 20:00:05.805936 28858 consensus_queue.cc:237] T fbcced7b903441278c70c4d3a1458be4 P a068ef6770f14874bd1acb01bce2c329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.809456 28551 catalog_manager.cc:5484] T 101b282620ad41b28af298749f60ec52 P dabdbfb59e8645c297a1696630198b6e reported cstate change: term changed from 0 to 1, leader changed from <none> to dabdbfb59e8645c297a1696630198b6e (127.27.217.193). New cstate: current_term: 1 leader_uuid: "dabdbfb59e8645c297a1696630198b6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } health_report { overall_health: UNKNOWN } } }
I20240501 20:00:05.825632 28849 tablet_bootstrap.cc:492] T 6a175ef3d4c94d459208ea5cfc31e9d8 P a068ef6770f14874bd1acb01bce2c329: No bootstrap required, opened a new log
I20240501 20:00:05.826154 28849 ts_tablet_manager.cc:1367] T 6a175ef3d4c94d459208ea5cfc31e9d8 P a068ef6770f14874bd1acb01bce2c329: Time spent bootstrapping tablet: real 0.085s user 0.015s sys 0.013s
I20240501 20:00:05.828720 28849 raft_consensus.cc:348] T 6a175ef3d4c94d459208ea5cfc31e9d8 P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.829336 28849 raft_consensus.cc:374] T 6a175ef3d4c94d459208ea5cfc31e9d8 P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:05.829735 28849 raft_consensus.cc:729] T 6a175ef3d4c94d459208ea5cfc31e9d8 P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a068ef6770f14874bd1acb01bce2c329, State: Initialized, Role: FOLLOWER
I20240501 20:00:05.829267 28550 catalog_manager.cc:5484] T fbcced7b903441278c70c4d3a1458be4 P a068ef6770f14874bd1acb01bce2c329 reported cstate change: term changed from 0 to 1, leader changed from <none> to a068ef6770f14874bd1acb01bce2c329 (127.27.217.194). New cstate: current_term: 1 leader_uuid: "a068ef6770f14874bd1acb01bce2c329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } health_report { overall_health: UNKNOWN } } }
I20240501 20:00:05.830271 28849 consensus_queue.cc:260] T 6a175ef3d4c94d459208ea5cfc31e9d8 P a068ef6770f14874bd1acb01bce2c329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.831678 28850 tablet_bootstrap.cc:492] T fbcced7b903441278c70c4d3a1458be4 P 3d7668f070e74286b38fda9d898f1acb: No bootstrap required, opened a new log
I20240501 20:00:05.832156 28850 ts_tablet_manager.cc:1367] T fbcced7b903441278c70c4d3a1458be4 P 3d7668f070e74286b38fda9d898f1acb: Time spent bootstrapping tablet: real 0.037s user 0.006s sys 0.004s
I20240501 20:00:05.832540 28849 ts_tablet_manager.cc:1398] T 6a175ef3d4c94d459208ea5cfc31e9d8 P a068ef6770f14874bd1acb01bce2c329: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20240501 20:00:05.833340 28849 tablet_bootstrap.cc:492] T 8bd1afda996d441b8d3eaa64dd36d1c7 P a068ef6770f14874bd1acb01bce2c329: Bootstrap starting.
I20240501 20:00:05.834582 28850 raft_consensus.cc:348] T fbcced7b903441278c70c4d3a1458be4 P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.835235 28850 raft_consensus.cc:374] T fbcced7b903441278c70c4d3a1458be4 P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:05.835489 28850 raft_consensus.cc:729] T fbcced7b903441278c70c4d3a1458be4 P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d7668f070e74286b38fda9d898f1acb, State: Initialized, Role: FOLLOWER
I20240501 20:00:05.836145 28850 consensus_queue.cc:260] T fbcced7b903441278c70c4d3a1458be4 P 3d7668f070e74286b38fda9d898f1acb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.838229 28850 ts_tablet_manager.cc:1398] T fbcced7b903441278c70c4d3a1458be4 P 3d7668f070e74286b38fda9d898f1acb: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240501 20:00:05.839020 28850 tablet_bootstrap.cc:492] T 676f538543c2491399b66b7db1490445 P 3d7668f070e74286b38fda9d898f1acb: Bootstrap starting.
I20240501 20:00:05.840106 28849 tablet_bootstrap.cc:654] T 8bd1afda996d441b8d3eaa64dd36d1c7 P a068ef6770f14874bd1acb01bce2c329: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:05.845466 28850 tablet_bootstrap.cc:654] T 676f538543c2491399b66b7db1490445 P 3d7668f070e74286b38fda9d898f1acb: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:05.851562 28877 raft_consensus.cc:482] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:05.852135 28877 raft_consensus.cc:504] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.854001 28877 leader_election.cc:288] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dabdbfb59e8645c297a1696630198b6e (127.27.217.193:46801), a068ef6770f14874bd1acb01bce2c329 (127.27.217.194:37307)
I20240501 20:00:05.855280 28654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a175ef3d4c94d459208ea5cfc31e9d8" candidate_uuid: "3d7668f070e74286b38fda9d898f1acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dabdbfb59e8645c297a1696630198b6e" is_pre_election: true
I20240501 20:00:05.855667 28849 tablet_bootstrap.cc:492] T 8bd1afda996d441b8d3eaa64dd36d1c7 P a068ef6770f14874bd1acb01bce2c329: No bootstrap required, opened a new log
I20240501 20:00:05.856300 28849 ts_tablet_manager.cc:1367] T 8bd1afda996d441b8d3eaa64dd36d1c7 P a068ef6770f14874bd1acb01bce2c329: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.019s
I20240501 20:00:05.856135 28654 raft_consensus.cc:2454] T 6a175ef3d4c94d459208ea5cfc31e9d8 P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d7668f070e74286b38fda9d898f1acb in term 0.
I20240501 20:00:05.857406 28768 leader_election.cc:302] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d7668f070e74286b38fda9d898f1acb, dabdbfb59e8645c297a1696630198b6e; no voters:
I20240501 20:00:05.857376 28728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a175ef3d4c94d459208ea5cfc31e9d8" candidate_uuid: "3d7668f070e74286b38fda9d898f1acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a068ef6770f14874bd1acb01bce2c329" is_pre_election: true
I20240501 20:00:05.858278 28863 raft_consensus.cc:482] T 676f538543c2491399b66b7db1490445 P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:05.858218 28728 raft_consensus.cc:2454] T 6a175ef3d4c94d459208ea5cfc31e9d8 P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d7668f070e74286b38fda9d898f1acb in term 0.
I20240501 20:00:05.858850 28863 raft_consensus.cc:504] T 676f538543c2491399b66b7db1490445 P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.858227 28877 raft_consensus.cc:2781] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:00:05.859669 28877 raft_consensus.cc:482] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:00:05.860141 28877 raft_consensus.cc:3037] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:05.860621 28863 leader_election.cc:288] T 676f538543c2491399b66b7db1490445 P dabdbfb59e8645c297a1696630198b6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a068ef6770f14874bd1acb01bce2c329 (127.27.217.194:37307), 3d7668f070e74286b38fda9d898f1acb (127.27.217.195:38723)
I20240501 20:00:05.862218 28803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "676f538543c2491399b66b7db1490445" candidate_uuid: "dabdbfb59e8645c297a1696630198b6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d7668f070e74286b38fda9d898f1acb" is_pre_election: true
W20240501 20:00:05.863400 28617 leader_election.cc:341] T 676f538543c2491399b66b7db1490445 P dabdbfb59e8645c297a1696630198b6e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d7668f070e74286b38fda9d898f1acb (127.27.217.195:38723): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:00:05.868988 28728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "676f538543c2491399b66b7db1490445" candidate_uuid: "dabdbfb59e8645c297a1696630198b6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a068ef6770f14874bd1acb01bce2c329" is_pre_election: true
I20240501 20:00:05.869752 28728 raft_consensus.cc:2454] T 676f538543c2491399b66b7db1490445 P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dabdbfb59e8645c297a1696630198b6e in term 0.
I20240501 20:00:05.884421 28617 leader_election.cc:302] T 676f538543c2491399b66b7db1490445 P dabdbfb59e8645c297a1696630198b6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a068ef6770f14874bd1acb01bce2c329, dabdbfb59e8645c297a1696630198b6e; no voters: 3d7668f070e74286b38fda9d898f1acb
I20240501 20:00:05.884356 28849 raft_consensus.cc:348] T 8bd1afda996d441b8d3eaa64dd36d1c7 P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.885213 28849 raft_consensus.cc:374] T 8bd1afda996d441b8d3eaa64dd36d1c7 P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:05.885306 28848 raft_consensus.cc:2781] T 676f538543c2491399b66b7db1490445 P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:00:05.885507 28849 raft_consensus.cc:729] T 8bd1afda996d441b8d3eaa64dd36d1c7 P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a068ef6770f14874bd1acb01bce2c329, State: Initialized, Role: FOLLOWER
I20240501 20:00:05.885737 28848 raft_consensus.cc:482] T 676f538543c2491399b66b7db1490445 P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:00:05.886126 28848 raft_consensus.cc:3037] T 676f538543c2491399b66b7db1490445 P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:05.886216 28849 consensus_queue.cc:260] T 8bd1afda996d441b8d3eaa64dd36d1c7 P a068ef6770f14874bd1acb01bce2c329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.886868 28877 raft_consensus.cc:504] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.887845 28849 ts_tablet_manager.cc:1398] T 8bd1afda996d441b8d3eaa64dd36d1c7 P a068ef6770f14874bd1acb01bce2c329: Time spent starting tablet: real 0.031s user 0.003s sys 0.000s
I20240501 20:00:05.893081 28654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a175ef3d4c94d459208ea5cfc31e9d8" candidate_uuid: "3d7668f070e74286b38fda9d898f1acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dabdbfb59e8645c297a1696630198b6e"
I20240501 20:00:05.893178 28877 leader_election.cc:288] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb [CANDIDATE]: Term 1 election: Requested vote from peers dabdbfb59e8645c297a1696630198b6e (127.27.217.193:46801), a068ef6770f14874bd1acb01bce2c329 (127.27.217.194:37307)
I20240501 20:00:05.894074 28654 raft_consensus.cc:3037] T 6a175ef3d4c94d459208ea5cfc31e9d8 P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:05.899735 28848 raft_consensus.cc:504] T 676f538543c2491399b66b7db1490445 P dabdbfb59e8645c297a1696630198b6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.902174 28729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a175ef3d4c94d459208ea5cfc31e9d8" candidate_uuid: "3d7668f070e74286b38fda9d898f1acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a068ef6770f14874bd1acb01bce2c329"
I20240501 20:00:05.904389 28729 raft_consensus.cc:3037] T 6a175ef3d4c94d459208ea5cfc31e9d8 P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:05.899818 28654 raft_consensus.cc:2454] T 6a175ef3d4c94d459208ea5cfc31e9d8 P dabdbfb59e8645c297a1696630198b6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d7668f070e74286b38fda9d898f1acb in term 1.
I20240501 20:00:05.909427 28848 leader_election.cc:288] T 676f538543c2491399b66b7db1490445 P dabdbfb59e8645c297a1696630198b6e [CANDIDATE]: Term 1 election: Requested vote from peers a068ef6770f14874bd1acb01bce2c329 (127.27.217.194:37307), 3d7668f070e74286b38fda9d898f1acb (127.27.217.195:38723)
I20240501 20:00:05.911478 28728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "676f538543c2491399b66b7db1490445" candidate_uuid: "dabdbfb59e8645c297a1696630198b6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a068ef6770f14874bd1acb01bce2c329"
I20240501 20:00:05.911875 28803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "676f538543c2491399b66b7db1490445" candidate_uuid: "dabdbfb59e8645c297a1696630198b6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d7668f070e74286b38fda9d898f1acb"
I20240501 20:00:05.911993 28728 raft_consensus.cc:3037] T 676f538543c2491399b66b7db1490445 P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Advancing to term 1
W20240501 20:00:05.913316 28617 leader_election.cc:341] T 676f538543c2491399b66b7db1490445 P dabdbfb59e8645c297a1696630198b6e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3d7668f070e74286b38fda9d898f1acb (127.27.217.195:38723): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:00:05.917369 28768 leader_election.cc:302] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d7668f070e74286b38fda9d898f1acb, dabdbfb59e8645c297a1696630198b6e; no voters:
I20240501 20:00:05.918283 28877 raft_consensus.cc:2781] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:00:05.918759 28877 raft_consensus.cc:686] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb [term 1 LEADER]: Becoming Leader. State: Replica: 3d7668f070e74286b38fda9d898f1acb, State: Running, Role: LEADER
I20240501 20:00:05.919559 28877 consensus_queue.cc:237] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.920447 28850 tablet_bootstrap.cc:492] T 676f538543c2491399b66b7db1490445 P 3d7668f070e74286b38fda9d898f1acb: No bootstrap required, opened a new log
I20240501 20:00:05.920881 28850 ts_tablet_manager.cc:1367] T 676f538543c2491399b66b7db1490445 P 3d7668f070e74286b38fda9d898f1acb: Time spent bootstrapping tablet: real 0.082s user 0.012s sys 0.023s
I20240501 20:00:05.922461 28728 raft_consensus.cc:2454] T 676f538543c2491399b66b7db1490445 P a068ef6770f14874bd1acb01bce2c329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dabdbfb59e8645c297a1696630198b6e in term 1.
I20240501 20:00:05.923630 28617 leader_election.cc:302] T 676f538543c2491399b66b7db1490445 P dabdbfb59e8645c297a1696630198b6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a068ef6770f14874bd1acb01bce2c329, dabdbfb59e8645c297a1696630198b6e; no voters: 3d7668f070e74286b38fda9d898f1acb
I20240501 20:00:05.910980 28863 consensus_queue.cc:1035] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20240501 20:00:05.924494 28729 raft_consensus.cc:2454] T 6a175ef3d4c94d459208ea5cfc31e9d8 P a068ef6770f14874bd1acb01bce2c329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d7668f070e74286b38fda9d898f1acb in term 1.
I20240501 20:00:05.923305 28850 raft_consensus.cc:348] T 676f538543c2491399b66b7db1490445 P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.925671 28899 raft_consensus.cc:2781] T 676f538543c2491399b66b7db1490445 P dabdbfb59e8645c297a1696630198b6e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:00:05.926302 28899 raft_consensus.cc:686] T 676f538543c2491399b66b7db1490445 P dabdbfb59e8645c297a1696630198b6e [term 1 LEADER]: Becoming Leader. State: Replica: dabdbfb59e8645c297a1696630198b6e, State: Running, Role: LEADER
I20240501 20:00:05.926064 28850 raft_consensus.cc:374] T 676f538543c2491399b66b7db1490445 P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:05.927121 28850 raft_consensus.cc:729] T 676f538543c2491399b66b7db1490445 P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d7668f070e74286b38fda9d898f1acb, State: Initialized, Role: FOLLOWER
I20240501 20:00:05.927755 28850 consensus_queue.cc:260] T 676f538543c2491399b66b7db1490445 P 3d7668f070e74286b38fda9d898f1acb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.927320 28899 consensus_queue.cc:237] T 676f538543c2491399b66b7db1490445 P dabdbfb59e8645c297a1696630198b6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:05.934574 28850 ts_tablet_manager.cc:1398] T 676f538543c2491399b66b7db1490445 P 3d7668f070e74286b38fda9d898f1acb: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20240501 20:00:05.928386 28550 catalog_manager.cc:5484] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d7668f070e74286b38fda9d898f1acb (127.27.217.195). New cstate: current_term: 1 leader_uuid: "3d7668f070e74286b38fda9d898f1acb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } health_report { overall_health: HEALTHY } } }
I20240501 20:00:05.955824 28550 catalog_manager.cc:5484] T 676f538543c2491399b66b7db1490445 P dabdbfb59e8645c297a1696630198b6e reported cstate change: term changed from 0 to 1, leader changed from <none> to dabdbfb59e8645c297a1696630198b6e (127.27.217.193). New cstate: current_term: 1 leader_uuid: "dabdbfb59e8645c297a1696630198b6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } health_report { overall_health: UNKNOWN } } }
I20240501 20:00:05.956750 28848 consensus_queue.cc:1035] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:00:05.986866 28877 raft_consensus.cc:482] T fbcced7b903441278c70c4d3a1458be4 P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:05.987531 28877 raft_consensus.cc:504] T fbcced7b903441278c70c4d3a1458be4 P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:06.001315 28879 consensus_queue.cc:1035] T 8a0820e7c49f40f89038e0af73567bac P 3d7668f070e74286b38fda9d898f1acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:00:06.036630 28879 raft_consensus.cc:482] T 676f538543c2491399b66b7db1490445 P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:06.037204 28879 raft_consensus.cc:504] T 676f538543c2491399b66b7db1490445 P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:06.047411 28858 raft_consensus.cc:482] T 8bd1afda996d441b8d3eaa64dd36d1c7 P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:06.048070 28858 raft_consensus.cc:504] T 8bd1afda996d441b8d3eaa64dd36d1c7 P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:06.057575 28859 consensus_queue.cc:1035] T 8a0820e7c49f40f89038e0af73567bac P 3d7668f070e74286b38fda9d898f1acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:00:06.085844 28653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd1afda996d441b8d3eaa64dd36d1c7" candidate_uuid: "a068ef6770f14874bd1acb01bce2c329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dabdbfb59e8645c297a1696630198b6e" is_pre_election: true
I20240501 20:00:06.086560 28653 raft_consensus.cc:2454] T 8bd1afda996d441b8d3eaa64dd36d1c7 P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a068ef6770f14874bd1acb01bce2c329 in term 0.
I20240501 20:00:06.086901 28877 leader_election.cc:288] T fbcced7b903441278c70c4d3a1458be4 P 3d7668f070e74286b38fda9d898f1acb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dabdbfb59e8645c297a1696630198b6e (127.27.217.193:46801), a068ef6770f14874bd1acb01bce2c329 (127.27.217.194:37307)
I20240501 20:00:06.090029 28728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbcced7b903441278c70c4d3a1458be4" candidate_uuid: "3d7668f070e74286b38fda9d898f1acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a068ef6770f14874bd1acb01bce2c329" is_pre_election: true
I20240501 20:00:06.090029 28802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd1afda996d441b8d3eaa64dd36d1c7" candidate_uuid: "a068ef6770f14874bd1acb01bce2c329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d7668f070e74286b38fda9d898f1acb" is_pre_election: true
I20240501 20:00:06.091667 28858 leader_election.cc:288] T 8bd1afda996d441b8d3eaa64dd36d1c7 P a068ef6770f14874bd1acb01bce2c329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dabdbfb59e8645c297a1696630198b6e (127.27.217.193:46801), 3d7668f070e74286b38fda9d898f1acb (127.27.217.195:38723)
I20240501 20:00:06.092535 28802 raft_consensus.cc:2454] T 8bd1afda996d441b8d3eaa64dd36d1c7 P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a068ef6770f14874bd1acb01bce2c329 in term 0.
I20240501 20:00:06.095515 28653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbcced7b903441278c70c4d3a1458be4" candidate_uuid: "3d7668f070e74286b38fda9d898f1acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dabdbfb59e8645c297a1696630198b6e" is_pre_election: true
I20240501 20:00:06.097982 28653 raft_consensus.cc:2379] T fbcced7b903441278c70c4d3a1458be4 P dabdbfb59e8645c297a1696630198b6e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d7668f070e74286b38fda9d898f1acb in current term 1: Already voted for candidate a068ef6770f14874bd1acb01bce2c329 in this term.
I20240501 20:00:06.098982 28768 leader_election.cc:302] T fbcced7b903441278c70c4d3a1458be4 P 3d7668f070e74286b38fda9d898f1acb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d7668f070e74286b38fda9d898f1acb; no voters: a068ef6770f14874bd1acb01bce2c329, dabdbfb59e8645c297a1696630198b6e
I20240501 20:00:06.099700 28877 raft_consensus.cc:3037] T fbcced7b903441278c70c4d3a1458be4 P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:06.091892 28652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "676f538543c2491399b66b7db1490445" candidate_uuid: "3d7668f070e74286b38fda9d898f1acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dabdbfb59e8645c297a1696630198b6e" is_pre_election: true
I20240501 20:00:06.100486 28848 raft_consensus.cc:482] T 14bfbbeb889d43218d5807488de4fed2 P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:06.100956 28848 raft_consensus.cc:504] T 14bfbbeb889d43218d5807488de4fed2 P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:06.092757 28694 leader_election.cc:302] T 8bd1afda996d441b8d3eaa64dd36d1c7 P a068ef6770f14874bd1acb01bce2c329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a068ef6770f14874bd1acb01bce2c329, dabdbfb59e8645c297a1696630198b6e; no voters:
I20240501 20:00:06.102910 28848 leader_election.cc:288] T 14bfbbeb889d43218d5807488de4fed2 P dabdbfb59e8645c297a1696630198b6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a068ef6770f14874bd1acb01bce2c329 (127.27.217.194:37307), 3d7668f070e74286b38fda9d898f1acb (127.27.217.195:38723)
I20240501 20:00:06.106098 28877 raft_consensus.cc:2726] T fbcced7b903441278c70c4d3a1458be4 P 3d7668f070e74286b38fda9d898f1acb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:00:06.107801 28802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14bfbbeb889d43218d5807488de4fed2" candidate_uuid: "dabdbfb59e8645c297a1696630198b6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d7668f070e74286b38fda9d898f1acb" is_pre_election: true
I20240501 20:00:06.108623 28802 raft_consensus.cc:2379] T 14bfbbeb889d43218d5807488de4fed2 P 3d7668f070e74286b38fda9d898f1acb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dabdbfb59e8645c297a1696630198b6e in current term 1: Already voted for candidate a068ef6770f14874bd1acb01bce2c329 in this term.
I20240501 20:00:06.108597 28879 leader_election.cc:288] T 676f538543c2491399b66b7db1490445 P 3d7668f070e74286b38fda9d898f1acb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dabdbfb59e8645c297a1696630198b6e (127.27.217.193:46801), a068ef6770f14874bd1acb01bce2c329 (127.27.217.194:37307)
I20240501 20:00:06.113883 28858 raft_consensus.cc:2781] T 8bd1afda996d441b8d3eaa64dd36d1c7 P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:00:06.114238 28858 raft_consensus.cc:482] T 8bd1afda996d441b8d3eaa64dd36d1c7 P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:00:06.114573 28858 raft_consensus.cc:3037] T 8bd1afda996d441b8d3eaa64dd36d1c7 P a068ef6770f14874bd1acb01bce2c329 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:06.120390 28729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "676f538543c2491399b66b7db1490445" candidate_uuid: "3d7668f070e74286b38fda9d898f1acb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a068ef6770f14874bd1acb01bce2c329" is_pre_election: true
I20240501 20:00:06.121155 28729 raft_consensus.cc:2379] T 676f538543c2491399b66b7db1490445 P a068ef6770f14874bd1acb01bce2c329 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d7668f070e74286b38fda9d898f1acb in current term 1: Already voted for candidate dabdbfb59e8645c297a1696630198b6e in this term.
I20240501 20:00:06.121999 28728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14bfbbeb889d43218d5807488de4fed2" candidate_uuid: "dabdbfb59e8645c297a1696630198b6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a068ef6770f14874bd1acb01bce2c329" is_pre_election: true
I20240501 20:00:06.122045 28766 leader_election.cc:302] T 676f538543c2491399b66b7db1490445 P 3d7668f070e74286b38fda9d898f1acb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d7668f070e74286b38fda9d898f1acb; no voters: a068ef6770f14874bd1acb01bce2c329, dabdbfb59e8645c297a1696630198b6e
I20240501 20:00:06.123381 28617 leader_election.cc:302] T 14bfbbeb889d43218d5807488de4fed2 P dabdbfb59e8645c297a1696630198b6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dabdbfb59e8645c297a1696630198b6e; no voters: 3d7668f070e74286b38fda9d898f1acb, a068ef6770f14874bd1acb01bce2c329
I20240501 20:00:06.124117 28848 raft_consensus.cc:3037] T 14bfbbeb889d43218d5807488de4fed2 P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:06.124543 28909 raft_consensus.cc:3037] T 676f538543c2491399b66b7db1490445 P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:06.125090 28858 raft_consensus.cc:504] T 8bd1afda996d441b8d3eaa64dd36d1c7 P a068ef6770f14874bd1acb01bce2c329 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:06.127631 28858 leader_election.cc:288] T 8bd1afda996d441b8d3eaa64dd36d1c7 P a068ef6770f14874bd1acb01bce2c329 [CANDIDATE]: Term 1 election: Requested vote from peers dabdbfb59e8645c297a1696630198b6e (127.27.217.193:46801), 3d7668f070e74286b38fda9d898f1acb (127.27.217.195:38723)
I20240501 20:00:06.128819 28802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd1afda996d441b8d3eaa64dd36d1c7" candidate_uuid: "a068ef6770f14874bd1acb01bce2c329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d7668f070e74286b38fda9d898f1acb"
I20240501 20:00:06.129377 28802 raft_consensus.cc:3037] T 8bd1afda996d441b8d3eaa64dd36d1c7 P 3d7668f070e74286b38fda9d898f1acb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:06.130299 28848 raft_consensus.cc:2726] T 14bfbbeb889d43218d5807488de4fed2 P dabdbfb59e8645c297a1696630198b6e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:00:06.131316 28652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd1afda996d441b8d3eaa64dd36d1c7" candidate_uuid: "a068ef6770f14874bd1acb01bce2c329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dabdbfb59e8645c297a1696630198b6e"
I20240501 20:00:06.131858 28652 raft_consensus.cc:3037] T 8bd1afda996d441b8d3eaa64dd36d1c7 P dabdbfb59e8645c297a1696630198b6e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:06.137569 28802 raft_consensus.cc:2454] T 8bd1afda996d441b8d3eaa64dd36d1c7 P 3d7668f070e74286b38fda9d898f1acb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a068ef6770f14874bd1acb01bce2c329 in term 1.
I20240501 20:00:06.138662 28692 leader_election.cc:302] T 8bd1afda996d441b8d3eaa64dd36d1c7 P a068ef6770f14874bd1acb01bce2c329 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d7668f070e74286b38fda9d898f1acb, a068ef6770f14874bd1acb01bce2c329; no voters:
I20240501 20:00:06.138917 28652 raft_consensus.cc:2454] T 8bd1afda996d441b8d3eaa64dd36d1c7 P dabdbfb59e8645c297a1696630198b6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a068ef6770f14874bd1acb01bce2c329 in term 1.
I20240501 20:00:06.140199 28858 raft_consensus.cc:2781] T 8bd1afda996d441b8d3eaa64dd36d1c7 P a068ef6770f14874bd1acb01bce2c329 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:00:06.140014 28877 consensus_queue.cc:1035] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:00:06.140782 28858 raft_consensus.cc:686] T 8bd1afda996d441b8d3eaa64dd36d1c7 P a068ef6770f14874bd1acb01bce2c329 [term 1 LEADER]: Becoming Leader. State: Replica: a068ef6770f14874bd1acb01bce2c329, State: Running, Role: LEADER
I20240501 20:00:06.143353 28858 consensus_queue.cc:237] T 8bd1afda996d441b8d3eaa64dd36d1c7 P a068ef6770f14874bd1acb01bce2c329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:06.145232 28909 raft_consensus.cc:2726] T 676f538543c2491399b66b7db1490445 P 3d7668f070e74286b38fda9d898f1acb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:00:06.152799 28879 consensus_queue.cc:1035] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:00:06.165925 28549 catalog_manager.cc:5484] T 8bd1afda996d441b8d3eaa64dd36d1c7 P a068ef6770f14874bd1acb01bce2c329 reported cstate change: term changed from 0 to 1, leader changed from <none> to a068ef6770f14874bd1acb01bce2c329 (127.27.217.194). New cstate: current_term: 1 leader_uuid: "a068ef6770f14874bd1acb01bce2c329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } health_report { overall_health: UNKNOWN } } }
I20240501 20:00:06.182847 28848 consensus_queue.cc:1035] T 101b282620ad41b28af298749f60ec52 P dabdbfb59e8645c297a1696630198b6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:00:06.209771 28863 consensus_queue.cc:1035] T 101b282620ad41b28af298749f60ec52 P dabdbfb59e8645c297a1696630198b6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:00:06.225246 28802 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6a175ef3d4c94d459208ea5cfc31e9d8"
dest_uuid: "3d7668f070e74286b38fda9d898f1acb"
mode: GRACEFUL
new_leader_uuid: "dabdbfb59e8645c297a1696630198b6e"
from {username='slave'} at 127.0.0.1:57390
I20240501 20:00:06.226117 28802 raft_consensus.cc:595] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb [term 1 LEADER]: Received request to transfer leadership to TS dabdbfb59e8645c297a1696630198b6e
I20240501 20:00:06.235921 28802 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8a0820e7c49f40f89038e0af73567bac"
dest_uuid: "3d7668f070e74286b38fda9d898f1acb"
mode: GRACEFUL
new_leader_uuid: "dabdbfb59e8645c297a1696630198b6e"
from {username='slave'} at 127.0.0.1:57390
I20240501 20:00:06.236572 28802 raft_consensus.cc:595] T 8a0820e7c49f40f89038e0af73567bac P 3d7668f070e74286b38fda9d898f1acb [term 1 LEADER]: Received request to transfer leadership to TS dabdbfb59e8645c297a1696630198b6e
I20240501 20:00:06.244566 28802 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "21dd29f1917a47f7ab496136a4c7ca07"
dest_uuid: "3d7668f070e74286b38fda9d898f1acb"
mode: GRACEFUL
new_leader_uuid: "a068ef6770f14874bd1acb01bce2c329"
from {username='slave'} at 127.0.0.1:57390
I20240501 20:00:06.245183 28802 raft_consensus.cc:595] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb [term 1 LEADER]: Received request to transfer leadership to TS a068ef6770f14874bd1acb01bce2c329
I20240501 20:00:06.259584 28858 consensus_queue.cc:1035] T 14bfbbeb889d43218d5807488de4fed2 P a068ef6770f14874bd1acb01bce2c329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:00:06.260898 28654 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "101b282620ad41b28af298749f60ec52"
dest_uuid: "dabdbfb59e8645c297a1696630198b6e"
mode: GRACEFUL
new_leader_uuid: "a068ef6770f14874bd1acb01bce2c329"
from {username='slave'} at 127.0.0.1:42564
I20240501 20:00:06.261487 28654 raft_consensus.cc:595] T 101b282620ad41b28af298749f60ec52 P dabdbfb59e8645c297a1696630198b6e [term 1 LEADER]: Received request to transfer leadership to TS a068ef6770f14874bd1acb01bce2c329
I20240501 20:00:06.272076 28858 consensus_queue.cc:1035] T fbcced7b903441278c70c4d3a1458be4 P a068ef6770f14874bd1acb01bce2c329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:00:06.276415 28858 consensus_queue.cc:1035] T 14bfbbeb889d43218d5807488de4fed2 P a068ef6770f14874bd1acb01bce2c329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:00:06.298460 28858 consensus_queue.cc:1035] T fbcced7b903441278c70c4d3a1458be4 P a068ef6770f14874bd1acb01bce2c329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:00:06.323213 28863 consensus_queue.cc:1035] T 676f538543c2491399b66b7db1490445 P dabdbfb59e8645c297a1696630198b6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:00:06.340452 28848 consensus_queue.cc:1035] T 676f538543c2491399b66b7db1490445 P dabdbfb59e8645c297a1696630198b6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:00:06.399159 28728 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "14bfbbeb889d43218d5807488de4fed2"
dest_uuid: "a068ef6770f14874bd1acb01bce2c329"
mode: GRACEFUL
new_leader_uuid: "3d7668f070e74286b38fda9d898f1acb"
from {username='slave'} at 127.0.0.1:48498
I20240501 20:00:06.399662 28728 raft_consensus.cc:595] T 14bfbbeb889d43218d5807488de4fed2 P a068ef6770f14874bd1acb01bce2c329 [term 1 LEADER]: Received request to transfer leadership to TS 3d7668f070e74286b38fda9d898f1acb
I20240501 20:00:06.450902 28877 consensus_queue.cc:1035] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:00:06.457656 28909 raft_consensus.cc:979] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb: : Instructing follower dabdbfb59e8645c297a1696630198b6e to start an election
I20240501 20:00:06.458107 28859 raft_consensus.cc:1067] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb [term 1 LEADER]: Signalling peer dabdbfb59e8645c297a1696630198b6e to start an election
I20240501 20:00:06.460839 28654 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6a175ef3d4c94d459208ea5cfc31e9d8"
dest_uuid: "dabdbfb59e8645c297a1696630198b6e"
from {username='slave'} at 127.0.0.1:42554
I20240501 20:00:06.461444 28654 raft_consensus.cc:482] T 6a175ef3d4c94d459208ea5cfc31e9d8 P dabdbfb59e8645c297a1696630198b6e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:00:06.461834 28654 raft_consensus.cc:3037] T 6a175ef3d4c94d459208ea5cfc31e9d8 P dabdbfb59e8645c297a1696630198b6e [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:06.463488 28859 consensus_queue.cc:1035] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:00:06.468240 28654 raft_consensus.cc:504] T 6a175ef3d4c94d459208ea5cfc31e9d8 P dabdbfb59e8645c297a1696630198b6e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:06.469338 28652 raft_consensus.cc:1226] T 6a175ef3d4c94d459208ea5cfc31e9d8 P dabdbfb59e8645c297a1696630198b6e [term 2 FOLLOWER]: Rejecting Update request from peer 3d7668f070e74286b38fda9d898f1acb for earlier term 1. Current term is 2. Ops: []
I20240501 20:00:06.473176 28654 leader_election.cc:288] T 6a175ef3d4c94d459208ea5cfc31e9d8 P dabdbfb59e8645c297a1696630198b6e [CANDIDATE]: Term 2 election: Requested vote from peers a068ef6770f14874bd1acb01bce2c329 (127.27.217.194:37307), 3d7668f070e74286b38fda9d898f1acb (127.27.217.195:38723)
I20240501 20:00:06.474350 28802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a175ef3d4c94d459208ea5cfc31e9d8" candidate_uuid: "dabdbfb59e8645c297a1696630198b6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3d7668f070e74286b38fda9d898f1acb"
I20240501 20:00:06.474965 28802 raft_consensus.cc:3032] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:00:06.475255 28802 raft_consensus.cc:729] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3d7668f070e74286b38fda9d898f1acb, State: Running, Role: LEADER
I20240501 20:00:06.476016 28802 consensus_queue.cc:260] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:06.477244 28802 raft_consensus.cc:3037] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:06.477725 28729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a175ef3d4c94d459208ea5cfc31e9d8" candidate_uuid: "dabdbfb59e8645c297a1696630198b6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a068ef6770f14874bd1acb01bce2c329"
I20240501 20:00:06.478106 28729 raft_consensus.cc:3037] T 6a175ef3d4c94d459208ea5cfc31e9d8 P a068ef6770f14874bd1acb01bce2c329 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:06.483357 28729 raft_consensus.cc:2454] T 6a175ef3d4c94d459208ea5cfc31e9d8 P a068ef6770f14874bd1acb01bce2c329 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dabdbfb59e8645c297a1696630198b6e in term 2.
I20240501 20:00:06.483505 28802 raft_consensus.cc:2454] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dabdbfb59e8645c297a1696630198b6e in term 2.
I20240501 20:00:06.484508 28617 leader_election.cc:302] T 6a175ef3d4c94d459208ea5cfc31e9d8 P dabdbfb59e8645c297a1696630198b6e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a068ef6770f14874bd1acb01bce2c329, dabdbfb59e8645c297a1696630198b6e; no voters:
I20240501 20:00:06.485303 28848 raft_consensus.cc:2781] T 6a175ef3d4c94d459208ea5cfc31e9d8 P dabdbfb59e8645c297a1696630198b6e [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:00:06.485752 28848 raft_consensus.cc:686] T 6a175ef3d4c94d459208ea5cfc31e9d8 P dabdbfb59e8645c297a1696630198b6e [term 2 LEADER]: Becoming Leader. State: Replica: dabdbfb59e8645c297a1696630198b6e, State: Running, Role: LEADER
I20240501 20:00:06.486527 28848 consensus_queue.cc:237] T 6a175ef3d4c94d459208ea5cfc31e9d8 P dabdbfb59e8645c297a1696630198b6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:06.492848 28548 catalog_manager.cc:5484] T 6a175ef3d4c94d459208ea5cfc31e9d8 P dabdbfb59e8645c297a1696630198b6e reported cstate change: term changed from 1 to 2, leader changed from 3d7668f070e74286b38fda9d898f1acb (127.27.217.195) to dabdbfb59e8645c297a1696630198b6e (127.27.217.193). New cstate: current_term: 2 leader_uuid: "dabdbfb59e8645c297a1696630198b6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } health_report { overall_health: UNKNOWN } } }
I20240501 20:00:06.620301 28879 raft_consensus.cc:979] T 8a0820e7c49f40f89038e0af73567bac P 3d7668f070e74286b38fda9d898f1acb: : Instructing follower dabdbfb59e8645c297a1696630198b6e to start an election
I20240501 20:00:06.620747 28859 raft_consensus.cc:1067] T 8a0820e7c49f40f89038e0af73567bac P 3d7668f070e74286b38fda9d898f1acb [term 1 LEADER]: Signalling peer dabdbfb59e8645c297a1696630198b6e to start an election
I20240501 20:00:06.622054 28652 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8a0820e7c49f40f89038e0af73567bac"
dest_uuid: "dabdbfb59e8645c297a1696630198b6e"
from {username='slave'} at 127.0.0.1:42554
I20240501 20:00:06.622484 28652 raft_consensus.cc:482] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:00:06.622742 28652 raft_consensus.cc:3037] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:06.626700 28652 raft_consensus.cc:504] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:06.628028 28652 leader_election.cc:288] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e [CANDIDATE]: Term 2 election: Requested vote from peers a068ef6770f14874bd1acb01bce2c329 (127.27.217.194:37307), 3d7668f070e74286b38fda9d898f1acb (127.27.217.195:38723)
I20240501 20:00:06.628902 28729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0820e7c49f40f89038e0af73567bac" candidate_uuid: "dabdbfb59e8645c297a1696630198b6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a068ef6770f14874bd1acb01bce2c329"
I20240501 20:00:06.629171 28802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0820e7c49f40f89038e0af73567bac" candidate_uuid: "dabdbfb59e8645c297a1696630198b6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3d7668f070e74286b38fda9d898f1acb"
I20240501 20:00:06.629438 28729 raft_consensus.cc:3037] T 8a0820e7c49f40f89038e0af73567bac P a068ef6770f14874bd1acb01bce2c329 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:06.629654 28802 raft_consensus.cc:3032] T 8a0820e7c49f40f89038e0af73567bac P 3d7668f070e74286b38fda9d898f1acb [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:00:06.629949 28802 raft_consensus.cc:729] T 8a0820e7c49f40f89038e0af73567bac P 3d7668f070e74286b38fda9d898f1acb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3d7668f070e74286b38fda9d898f1acb, State: Running, Role: LEADER
I20240501 20:00:06.630681 28802 consensus_queue.cc:260] T 8a0820e7c49f40f89038e0af73567bac P 3d7668f070e74286b38fda9d898f1acb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:06.631842 28802 raft_consensus.cc:3037] T 8a0820e7c49f40f89038e0af73567bac P 3d7668f070e74286b38fda9d898f1acb [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:06.635111 28729 raft_consensus.cc:2454] T 8a0820e7c49f40f89038e0af73567bac P a068ef6770f14874bd1acb01bce2c329 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dabdbfb59e8645c297a1696630198b6e in term 2.
I20240501 20:00:06.636157 28617 leader_election.cc:302] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a068ef6770f14874bd1acb01bce2c329, dabdbfb59e8645c297a1696630198b6e; no voters:
I20240501 20:00:06.636940 28848 raft_consensus.cc:2781] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:00:06.637405 28848 raft_consensus.cc:686] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e [term 2 LEADER]: Becoming Leader. State: Replica: dabdbfb59e8645c297a1696630198b6e, State: Running, Role: LEADER
I20240501 20:00:06.637465 28802 raft_consensus.cc:2454] T 8a0820e7c49f40f89038e0af73567bac P 3d7668f070e74286b38fda9d898f1acb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dabdbfb59e8645c297a1696630198b6e in term 2.
I20240501 20:00:06.638190 28848 consensus_queue.cc:237] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:06.645164 28548 catalog_manager.cc:5484] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e reported cstate change: term changed from 1 to 2, leader changed from 3d7668f070e74286b38fda9d898f1acb (127.27.217.195) to dabdbfb59e8645c297a1696630198b6e (127.27.217.193). New cstate: current_term: 2 leader_uuid: "dabdbfb59e8645c297a1696630198b6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } health_report { overall_health: UNKNOWN } } }
I20240501 20:00:06.675700 28859 raft_consensus.cc:979] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb: : Instructing follower a068ef6770f14874bd1acb01bce2c329 to start an election
I20240501 20:00:06.676272 28909 raft_consensus.cc:1067] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb [term 1 LEADER]: Signalling peer a068ef6770f14874bd1acb01bce2c329 to start an election
I20240501 20:00:06.677006 28927 consensus_queue.cc:1035] T 8bd1afda996d441b8d3eaa64dd36d1c7 P a068ef6770f14874bd1acb01bce2c329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:00:06.679340 28729 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "21dd29f1917a47f7ab496136a4c7ca07"
dest_uuid: "a068ef6770f14874bd1acb01bce2c329"
from {username='slave'} at 127.0.0.1:48486
I20240501 20:00:06.679869 28729 raft_consensus.cc:482] T 21dd29f1917a47f7ab496136a4c7ca07 P a068ef6770f14874bd1acb01bce2c329 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:00:06.680274 28729 raft_consensus.cc:3037] T 21dd29f1917a47f7ab496136a4c7ca07 P a068ef6770f14874bd1acb01bce2c329 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:06.686023 28729 raft_consensus.cc:504] T 21dd29f1917a47f7ab496136a4c7ca07 P a068ef6770f14874bd1acb01bce2c329 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:06.687055 28927 consensus_queue.cc:1035] T 8bd1afda996d441b8d3eaa64dd36d1c7 P a068ef6770f14874bd1acb01bce2c329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:00:06.697973 28729 leader_election.cc:288] T 21dd29f1917a47f7ab496136a4c7ca07 P a068ef6770f14874bd1acb01bce2c329 [CANDIDATE]: Term 2 election: Requested vote from peers dabdbfb59e8645c297a1696630198b6e (127.27.217.193:46801), 3d7668f070e74286b38fda9d898f1acb (127.27.217.195:38723)
I20240501 20:00:06.699519 28803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21dd29f1917a47f7ab496136a4c7ca07" candidate_uuid: "a068ef6770f14874bd1acb01bce2c329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3d7668f070e74286b38fda9d898f1acb"
I20240501 20:00:06.700213 28803 raft_consensus.cc:3032] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:00:06.700569 28803 raft_consensus.cc:729] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3d7668f070e74286b38fda9d898f1acb, State: Running, Role: LEADER
I20240501 20:00:06.701288 28803 consensus_queue.cc:260] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:06.702586 28803 raft_consensus.cc:3037] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:06.707230 28803 raft_consensus.cc:2454] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a068ef6770f14874bd1acb01bce2c329 in term 2.
I20240501 20:00:06.708398 28692 leader_election.cc:302] T 21dd29f1917a47f7ab496136a4c7ca07 P a068ef6770f14874bd1acb01bce2c329 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d7668f070e74286b38fda9d898f1acb, a068ef6770f14874bd1acb01bce2c329; no voters:
I20240501 20:00:06.709231 28924 raft_consensus.cc:2781] T 21dd29f1917a47f7ab496136a4c7ca07 P a068ef6770f14874bd1acb01bce2c329 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:00:06.709671 28924 raft_consensus.cc:686] T 21dd29f1917a47f7ab496136a4c7ca07 P a068ef6770f14874bd1acb01bce2c329 [term 2 LEADER]: Becoming Leader. State: Replica: a068ef6770f14874bd1acb01bce2c329, State: Running, Role: LEADER
I20240501 20:00:06.710237 28924 consensus_queue.cc:237] T 21dd29f1917a47f7ab496136a4c7ca07 P a068ef6770f14874bd1acb01bce2c329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:06.716392 28652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21dd29f1917a47f7ab496136a4c7ca07" candidate_uuid: "a068ef6770f14874bd1acb01bce2c329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dabdbfb59e8645c297a1696630198b6e"
I20240501 20:00:06.717001 28652 raft_consensus.cc:3037] T 21dd29f1917a47f7ab496136a4c7ca07 P dabdbfb59e8645c297a1696630198b6e [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:06.718144 28550 catalog_manager.cc:5484] T 21dd29f1917a47f7ab496136a4c7ca07 P a068ef6770f14874bd1acb01bce2c329 reported cstate change: term changed from 1 to 2, leader changed from 3d7668f070e74286b38fda9d898f1acb (127.27.217.195) to a068ef6770f14874bd1acb01bce2c329 (127.27.217.194). New cstate: current_term: 2 leader_uuid: "a068ef6770f14874bd1acb01bce2c329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } health_report { overall_health: UNKNOWN } } }
I20240501 20:00:06.724093 28652 raft_consensus.cc:2454] T 21dd29f1917a47f7ab496136a4c7ca07 P dabdbfb59e8645c297a1696630198b6e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a068ef6770f14874bd1acb01bce2c329 in term 2.
I20240501 20:00:06.766831 28899 raft_consensus.cc:979] T 101b282620ad41b28af298749f60ec52 P dabdbfb59e8645c297a1696630198b6e: : Instructing follower a068ef6770f14874bd1acb01bce2c329 to start an election
I20240501 20:00:06.767316 28848 raft_consensus.cc:1067] T 101b282620ad41b28af298749f60ec52 P dabdbfb59e8645c297a1696630198b6e [term 1 LEADER]: Signalling peer a068ef6770f14874bd1acb01bce2c329 to start an election
I20240501 20:00:06.768828 28729 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "101b282620ad41b28af298749f60ec52"
dest_uuid: "a068ef6770f14874bd1acb01bce2c329"
from {username='slave'} at 127.0.0.1:48472
I20240501 20:00:06.769387 28729 raft_consensus.cc:482] T 101b282620ad41b28af298749f60ec52 P a068ef6770f14874bd1acb01bce2c329 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:00:06.769739 28729 raft_consensus.cc:3037] T 101b282620ad41b28af298749f60ec52 P a068ef6770f14874bd1acb01bce2c329 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:06.774123 28729 raft_consensus.cc:504] T 101b282620ad41b28af298749f60ec52 P a068ef6770f14874bd1acb01bce2c329 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:06.776177 28729 leader_election.cc:288] T 101b282620ad41b28af298749f60ec52 P a068ef6770f14874bd1acb01bce2c329 [CANDIDATE]: Term 2 election: Requested vote from peers dabdbfb59e8645c297a1696630198b6e (127.27.217.193:46801), 3d7668f070e74286b38fda9d898f1acb (127.27.217.195:38723)
I20240501 20:00:06.776461 28803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "101b282620ad41b28af298749f60ec52" candidate_uuid: "a068ef6770f14874bd1acb01bce2c329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3d7668f070e74286b38fda9d898f1acb"
I20240501 20:00:06.776203 28652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "101b282620ad41b28af298749f60ec52" candidate_uuid: "a068ef6770f14874bd1acb01bce2c329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dabdbfb59e8645c297a1696630198b6e"
I20240501 20:00:06.777257 28803 raft_consensus.cc:3037] T 101b282620ad41b28af298749f60ec52 P 3d7668f070e74286b38fda9d898f1acb [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:06.777518 28652 raft_consensus.cc:3032] T 101b282620ad41b28af298749f60ec52 P dabdbfb59e8645c297a1696630198b6e [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:00:06.777873 28652 raft_consensus.cc:729] T 101b282620ad41b28af298749f60ec52 P dabdbfb59e8645c297a1696630198b6e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dabdbfb59e8645c297a1696630198b6e, State: Running, Role: LEADER
I20240501 20:00:06.778687 28652 consensus_queue.cc:260] T 101b282620ad41b28af298749f60ec52 P dabdbfb59e8645c297a1696630198b6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:06.779858 28652 raft_consensus.cc:3037] T 101b282620ad41b28af298749f60ec52 P dabdbfb59e8645c297a1696630198b6e [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:06.783365 28858 raft_consensus.cc:979] T 14bfbbeb889d43218d5807488de4fed2 P a068ef6770f14874bd1acb01bce2c329: : Instructing follower 3d7668f070e74286b38fda9d898f1acb to start an election
I20240501 20:00:06.783797 28858 raft_consensus.cc:1067] T 14bfbbeb889d43218d5807488de4fed2 P a068ef6770f14874bd1acb01bce2c329 [term 1 LEADER]: Signalling peer 3d7668f070e74286b38fda9d898f1acb to start an election
I20240501 20:00:06.785125 28802 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "14bfbbeb889d43218d5807488de4fed2"
dest_uuid: "3d7668f070e74286b38fda9d898f1acb"
from {username='slave'} at 127.0.0.1:57378
I20240501 20:00:06.785638 28802 raft_consensus.cc:482] T 14bfbbeb889d43218d5807488de4fed2 P 3d7668f070e74286b38fda9d898f1acb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:00:06.785965 28802 raft_consensus.cc:3037] T 14bfbbeb889d43218d5807488de4fed2 P 3d7668f070e74286b38fda9d898f1acb [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:06.786067 28652 raft_consensus.cc:2454] T 101b282620ad41b28af298749f60ec52 P dabdbfb59e8645c297a1696630198b6e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a068ef6770f14874bd1acb01bce2c329 in term 2.
I20240501 20:00:06.787202 28694 leader_election.cc:302] T 101b282620ad41b28af298749f60ec52 P a068ef6770f14874bd1acb01bce2c329 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a068ef6770f14874bd1acb01bce2c329, dabdbfb59e8645c297a1696630198b6e; no voters:
I20240501 20:00:06.787308 28803 raft_consensus.cc:2454] T 101b282620ad41b28af298749f60ec52 P 3d7668f070e74286b38fda9d898f1acb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a068ef6770f14874bd1acb01bce2c329 in term 2.
I20240501 20:00:06.788095 28924 raft_consensus.cc:2781] T 101b282620ad41b28af298749f60ec52 P a068ef6770f14874bd1acb01bce2c329 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:00:06.788540 28924 raft_consensus.cc:686] T 101b282620ad41b28af298749f60ec52 P a068ef6770f14874bd1acb01bce2c329 [term 2 LEADER]: Becoming Leader. State: Replica: a068ef6770f14874bd1acb01bce2c329, State: Running, Role: LEADER
I20240501 20:00:06.789302 28924 consensus_queue.cc:237] T 101b282620ad41b28af298749f60ec52 P a068ef6770f14874bd1acb01bce2c329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:06.793861 28802 raft_consensus.cc:504] T 14bfbbeb889d43218d5807488de4fed2 P 3d7668f070e74286b38fda9d898f1acb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:06.795794 28802 leader_election.cc:288] T 14bfbbeb889d43218d5807488de4fed2 P 3d7668f070e74286b38fda9d898f1acb [CANDIDATE]: Term 2 election: Requested vote from peers dabdbfb59e8645c297a1696630198b6e (127.27.217.193:46801), a068ef6770f14874bd1acb01bce2c329 (127.27.217.194:37307)
I20240501 20:00:06.796705 28652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14bfbbeb889d43218d5807488de4fed2" candidate_uuid: "3d7668f070e74286b38fda9d898f1acb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dabdbfb59e8645c297a1696630198b6e"
I20240501 20:00:06.797570 28652 raft_consensus.cc:3037] T 14bfbbeb889d43218d5807488de4fed2 P dabdbfb59e8645c297a1696630198b6e [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:06.797587 28729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14bfbbeb889d43218d5807488de4fed2" candidate_uuid: "3d7668f070e74286b38fda9d898f1acb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a068ef6770f14874bd1acb01bce2c329"
I20240501 20:00:06.799427 28729 raft_consensus.cc:3032] T 14bfbbeb889d43218d5807488de4fed2 P a068ef6770f14874bd1acb01bce2c329 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:00:06.799728 28729 raft_consensus.cc:729] T 14bfbbeb889d43218d5807488de4fed2 P a068ef6770f14874bd1acb01bce2c329 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a068ef6770f14874bd1acb01bce2c329, State: Running, Role: LEADER
I20240501 20:00:06.799031 28548 catalog_manager.cc:5484] T 101b282620ad41b28af298749f60ec52 P a068ef6770f14874bd1acb01bce2c329 reported cstate change: term changed from 1 to 2, leader changed from dabdbfb59e8645c297a1696630198b6e (127.27.217.193) to a068ef6770f14874bd1acb01bce2c329 (127.27.217.194). New cstate: current_term: 2 leader_uuid: "a068ef6770f14874bd1acb01bce2c329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } health_report { overall_health: UNKNOWN } } }
I20240501 20:00:06.800434 28729 consensus_queue.cc:260] T 14bfbbeb889d43218d5807488de4fed2 P a068ef6770f14874bd1acb01bce2c329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:06.801668 28729 raft_consensus.cc:3037] T 14bfbbeb889d43218d5807488de4fed2 P a068ef6770f14874bd1acb01bce2c329 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:06.803596 28652 raft_consensus.cc:2454] T 14bfbbeb889d43218d5807488de4fed2 P dabdbfb59e8645c297a1696630198b6e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d7668f070e74286b38fda9d898f1acb in term 2.
I20240501 20:00:06.804620 28768 leader_election.cc:302] T 14bfbbeb889d43218d5807488de4fed2 P 3d7668f070e74286b38fda9d898f1acb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d7668f070e74286b38fda9d898f1acb, dabdbfb59e8645c297a1696630198b6e; no voters:
I20240501 20:00:06.805411 28859 raft_consensus.cc:2781] T 14bfbbeb889d43218d5807488de4fed2 P 3d7668f070e74286b38fda9d898f1acb [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:00:06.805838 28859 raft_consensus.cc:686] T 14bfbbeb889d43218d5807488de4fed2 P 3d7668f070e74286b38fda9d898f1acb [term 2 LEADER]: Becoming Leader. State: Replica: 3d7668f070e74286b38fda9d898f1acb, State: Running, Role: LEADER
I20240501 20:00:06.806620 28859 consensus_queue.cc:237] T 14bfbbeb889d43218d5807488de4fed2 P 3d7668f070e74286b38fda9d898f1acb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:06.807585 28729 raft_consensus.cc:2454] T 14bfbbeb889d43218d5807488de4fed2 P a068ef6770f14874bd1acb01bce2c329 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d7668f070e74286b38fda9d898f1acb in term 2.
I20240501 20:00:06.813874 28548 catalog_manager.cc:5484] T 14bfbbeb889d43218d5807488de4fed2 P 3d7668f070e74286b38fda9d898f1acb reported cstate change: term changed from 1 to 2, leader changed from a068ef6770f14874bd1acb01bce2c329 (127.27.217.194) to 3d7668f070e74286b38fda9d898f1acb (127.27.217.195). New cstate: current_term: 2 leader_uuid: "3d7668f070e74286b38fda9d898f1acb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } health_report { overall_health: HEALTHY } } }
I20240501 20:00:07.004907 28802 raft_consensus.cc:1261] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb [term 2 FOLLOWER]: Refusing update from remote peer dabdbfb59e8645c297a1696630198b6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:00:07.006409 28848 consensus_queue.cc:1035] T 6a175ef3d4c94d459208ea5cfc31e9d8 P dabdbfb59e8645c297a1696630198b6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:00:07.014879 28729 raft_consensus.cc:1261] T 6a175ef3d4c94d459208ea5cfc31e9d8 P a068ef6770f14874bd1acb01bce2c329 [term 2 FOLLOWER]: Refusing update from remote peer dabdbfb59e8645c297a1696630198b6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:00:07.016578 28899 consensus_queue.cc:1035] T 6a175ef3d4c94d459208ea5cfc31e9d8 P dabdbfb59e8645c297a1696630198b6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:00:07.097442 28652 raft_consensus.cc:1261] T 21dd29f1917a47f7ab496136a4c7ca07 P dabdbfb59e8645c297a1696630198b6e [term 2 FOLLOWER]: Refusing update from remote peer a068ef6770f14874bd1acb01bce2c329: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:00:07.098686 28924 consensus_queue.cc:1035] T 21dd29f1917a47f7ab496136a4c7ca07 P a068ef6770f14874bd1acb01bce2c329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:00:07.107213 28802 raft_consensus.cc:1261] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb [term 2 FOLLOWER]: Refusing update from remote peer a068ef6770f14874bd1acb01bce2c329: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:00:07.108573 28927 consensus_queue.cc:1035] T 21dd29f1917a47f7ab496136a4c7ca07 P a068ef6770f14874bd1acb01bce2c329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:00:07.202296 28652 raft_consensus.cc:1261] T 14bfbbeb889d43218d5807488de4fed2 P dabdbfb59e8645c297a1696630198b6e [term 2 FOLLOWER]: Refusing update from remote peer 3d7668f070e74286b38fda9d898f1acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:00:07.203436 28909 consensus_queue.cc:1035] T 14bfbbeb889d43218d5807488de4fed2 P 3d7668f070e74286b38fda9d898f1acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:00:07.212708 28729 raft_consensus.cc:1261] T 14bfbbeb889d43218d5807488de4fed2 P a068ef6770f14874bd1acb01bce2c329 [term 2 FOLLOWER]: Refusing update from remote peer 3d7668f070e74286b38fda9d898f1acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:00:07.214923 28909 consensus_queue.cc:1035] T 14bfbbeb889d43218d5807488de4fed2 P 3d7668f070e74286b38fda9d898f1acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:00:07.241334 28729 raft_consensus.cc:1261] T 8a0820e7c49f40f89038e0af73567bac P a068ef6770f14874bd1acb01bce2c329 [term 2 FOLLOWER]: Refusing update from remote peer dabdbfb59e8645c297a1696630198b6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:00:07.242410 28899 consensus_queue.cc:1035] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:00:07.249465 28802 raft_consensus.cc:1261] T 8a0820e7c49f40f89038e0af73567bac P 3d7668f070e74286b38fda9d898f1acb [term 2 FOLLOWER]: Refusing update from remote peer dabdbfb59e8645c297a1696630198b6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:00:07.250744 28848 consensus_queue.cc:1035] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:00:07.274708 28652 raft_consensus.cc:1261] T 101b282620ad41b28af298749f60ec52 P dabdbfb59e8645c297a1696630198b6e [term 2 FOLLOWER]: Refusing update from remote peer a068ef6770f14874bd1acb01bce2c329: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:00:07.275815 28927 consensus_queue.cc:1035] T 101b282620ad41b28af298749f60ec52 P a068ef6770f14874bd1acb01bce2c329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:00:07.283110 28802 raft_consensus.cc:1261] T 101b282620ad41b28af298749f60ec52 P 3d7668f070e74286b38fda9d898f1acb [term 2 FOLLOWER]: Refusing update from remote peer a068ef6770f14874bd1acb01bce2c329: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:00:07.284440 28858 consensus_queue.cc:1035] T 101b282620ad41b28af298749f60ec52 P a068ef6770f14874bd1acb01bce2c329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:00:09.422329 28519 auto_leader_rebalancer-test.cc:261] The leader distribution is
I20240501 20:00:09.423065 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:09.442574 28728 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "21dd29f1917a47f7ab496136a4c7ca07"
dest_uuid: "a068ef6770f14874bd1acb01bce2c329"
mode: GRACEFUL
new_leader_uuid: "3d7668f070e74286b38fda9d898f1acb"
from {username='slave'} at 127.0.0.1:48506
I20240501 20:00:09.443101 28728 raft_consensus.cc:595] T 21dd29f1917a47f7ab496136a4c7ca07 P a068ef6770f14874bd1acb01bce2c329 [term 2 LEADER]: Received request to transfer leadership to TS 3d7668f070e74286b38fda9d898f1acb
I20240501 20:00:09.453472 28942 raft_consensus.cc:979] T 21dd29f1917a47f7ab496136a4c7ca07 P a068ef6770f14874bd1acb01bce2c329: : Instructing follower 3d7668f070e74286b38fda9d898f1acb to start an election
I20240501 20:00:09.453642 28652 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6a175ef3d4c94d459208ea5cfc31e9d8"
dest_uuid: "dabdbfb59e8645c297a1696630198b6e"
mode: GRACEFUL
new_leader_uuid: "3d7668f070e74286b38fda9d898f1acb"
from {username='slave'} at 127.0.0.1:42566
I20240501 20:00:09.453995 28924 raft_consensus.cc:1067] T 21dd29f1917a47f7ab496136a4c7ca07 P a068ef6770f14874bd1acb01bce2c329 [term 2 LEADER]: Signalling peer 3d7668f070e74286b38fda9d898f1acb to start an election
I20240501 20:00:09.454260 28652 raft_consensus.cc:595] T 6a175ef3d4c94d459208ea5cfc31e9d8 P dabdbfb59e8645c297a1696630198b6e [term 2 LEADER]: Received request to transfer leadership to TS 3d7668f070e74286b38fda9d898f1acb
I20240501 20:00:09.455401 28802 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "21dd29f1917a47f7ab496136a4c7ca07"
dest_uuid: "3d7668f070e74286b38fda9d898f1acb"
from {username='slave'} at 127.0.0.1:57378
I20240501 20:00:09.455651 28519 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240501 20:00:09.455968 28802 raft_consensus.cc:482] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:00:09.456250 28519 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:09.456385 28802 raft_consensus.cc:3037] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:00:09.461740 28802 raft_consensus.cc:504] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:09.463135 28802 leader_election.cc:288] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb [CANDIDATE]: Term 3 election: Requested vote from peers dabdbfb59e8645c297a1696630198b6e (127.27.217.193:46801), a068ef6770f14874bd1acb01bce2c329 (127.27.217.194:37307)
I20240501 20:00:09.463989 28652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21dd29f1917a47f7ab496136a4c7ca07" candidate_uuid: "3d7668f070e74286b38fda9d898f1acb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "dabdbfb59e8645c297a1696630198b6e"
I20240501 20:00:09.464301 28728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21dd29f1917a47f7ab496136a4c7ca07" candidate_uuid: "3d7668f070e74286b38fda9d898f1acb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a068ef6770f14874bd1acb01bce2c329"
I20240501 20:00:09.464494 28652 raft_consensus.cc:3037] T 21dd29f1917a47f7ab496136a4c7ca07 P dabdbfb59e8645c297a1696630198b6e [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:00:09.464804 28728 raft_consensus.cc:3032] T 21dd29f1917a47f7ab496136a4c7ca07 P a068ef6770f14874bd1acb01bce2c329 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:00:09.465071 28728 raft_consensus.cc:729] T 21dd29f1917a47f7ab496136a4c7ca07 P a068ef6770f14874bd1acb01bce2c329 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a068ef6770f14874bd1acb01bce2c329, State: Running, Role: LEADER
I20240501 20:00:09.465687 28728 consensus_queue.cc:260] T 21dd29f1917a47f7ab496136a4c7ca07 P a068ef6770f14874bd1acb01bce2c329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:09.466912 28728 raft_consensus.cc:3037] T 21dd29f1917a47f7ab496136a4c7ca07 P a068ef6770f14874bd1acb01bce2c329 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:00:09.470136 28652 raft_consensus.cc:2454] T 21dd29f1917a47f7ab496136a4c7ca07 P dabdbfb59e8645c297a1696630198b6e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d7668f070e74286b38fda9d898f1acb in term 3.
I20240501 20:00:09.471282 28768 leader_election.cc:302] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d7668f070e74286b38fda9d898f1acb, dabdbfb59e8645c297a1696630198b6e; no voters:
I20240501 20:00:09.472100 28938 raft_consensus.cc:2781] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:00:09.472806 28728 raft_consensus.cc:2454] T 21dd29f1917a47f7ab496136a4c7ca07 P a068ef6770f14874bd1acb01bce2c329 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d7668f070e74286b38fda9d898f1acb in term 3.
I20240501 20:00:09.473331 28938 raft_consensus.cc:686] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb [term 3 LEADER]: Becoming Leader. State: Replica: 3d7668f070e74286b38fda9d898f1acb, State: Running, Role: LEADER
I20240501 20:00:09.474110 28938 consensus_queue.cc:237] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:09.480221 28548 catalog_manager.cc:5484] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb reported cstate change: term changed from 2 to 3, leader changed from a068ef6770f14874bd1acb01bce2c329 (127.27.217.194) to 3d7668f070e74286b38fda9d898f1acb (127.27.217.195). New cstate: current_term: 3 leader_uuid: "3d7668f070e74286b38fda9d898f1acb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } health_report { overall_health: HEALTHY } } }
I20240501 20:00:09.850142 28949 raft_consensus.cc:979] T 6a175ef3d4c94d459208ea5cfc31e9d8 P dabdbfb59e8645c297a1696630198b6e: : Instructing follower 3d7668f070e74286b38fda9d898f1acb to start an election
I20240501 20:00:09.850579 28848 raft_consensus.cc:1067] T 6a175ef3d4c94d459208ea5cfc31e9d8 P dabdbfb59e8645c297a1696630198b6e [term 2 LEADER]: Signalling peer 3d7668f070e74286b38fda9d898f1acb to start an election
I20240501 20:00:09.851858 28802 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6a175ef3d4c94d459208ea5cfc31e9d8"
dest_uuid: "3d7668f070e74286b38fda9d898f1acb"
from {username='slave'} at 127.0.0.1:57366
I20240501 20:00:09.852321 28802 raft_consensus.cc:482] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:00:09.852594 28802 raft_consensus.cc:3037] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:00:09.856990 28802 raft_consensus.cc:504] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:09.858546 28802 leader_election.cc:288] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb [CANDIDATE]: Term 3 election: Requested vote from peers dabdbfb59e8645c297a1696630198b6e (127.27.217.193:46801), a068ef6770f14874bd1acb01bce2c329 (127.27.217.194:37307)
I20240501 20:00:09.859333 28652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a175ef3d4c94d459208ea5cfc31e9d8" candidate_uuid: "3d7668f070e74286b38fda9d898f1acb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "dabdbfb59e8645c297a1696630198b6e"
I20240501 20:00:09.859540 28728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a175ef3d4c94d459208ea5cfc31e9d8" candidate_uuid: "3d7668f070e74286b38fda9d898f1acb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a068ef6770f14874bd1acb01bce2c329"
I20240501 20:00:09.859966 28652 raft_consensus.cc:3032] T 6a175ef3d4c94d459208ea5cfc31e9d8 P dabdbfb59e8645c297a1696630198b6e [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:00:09.860085 28728 raft_consensus.cc:3037] T 6a175ef3d4c94d459208ea5cfc31e9d8 P a068ef6770f14874bd1acb01bce2c329 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:00:09.860385 28652 raft_consensus.cc:729] T 6a175ef3d4c94d459208ea5cfc31e9d8 P dabdbfb59e8645c297a1696630198b6e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: dabdbfb59e8645c297a1696630198b6e, State: Running, Role: LEADER
I20240501 20:00:09.861146 28652 consensus_queue.cc:260] T 6a175ef3d4c94d459208ea5cfc31e9d8 P dabdbfb59e8645c297a1696630198b6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:09.862370 28652 raft_consensus.cc:3037] T 6a175ef3d4c94d459208ea5cfc31e9d8 P dabdbfb59e8645c297a1696630198b6e [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:00:09.866422 28728 raft_consensus.cc:2454] T 6a175ef3d4c94d459208ea5cfc31e9d8 P a068ef6770f14874bd1acb01bce2c329 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d7668f070e74286b38fda9d898f1acb in term 3.
I20240501 20:00:09.867753 28766 leader_election.cc:302] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d7668f070e74286b38fda9d898f1acb, a068ef6770f14874bd1acb01bce2c329; no voters:
I20240501 20:00:09.868553 28938 raft_consensus.cc:2781] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:00:09.868619 28652 raft_consensus.cc:2454] T 6a175ef3d4c94d459208ea5cfc31e9d8 P dabdbfb59e8645c297a1696630198b6e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d7668f070e74286b38fda9d898f1acb in term 3.
I20240501 20:00:09.869108 28938 raft_consensus.cc:686] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb [term 3 LEADER]: Becoming Leader. State: Replica: 3d7668f070e74286b38fda9d898f1acb, State: Running, Role: LEADER
I20240501 20:00:09.869360 28654 raft_consensus.cc:1261] T 21dd29f1917a47f7ab496136a4c7ca07 P dabdbfb59e8645c297a1696630198b6e [term 3 FOLLOWER]: Refusing update from remote peer 3d7668f070e74286b38fda9d898f1acb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 20:00:09.869866 28938 consensus_queue.cc:237] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:09.870849 28943 consensus_queue.cc:1035] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:00:09.884858 28728 raft_consensus.cc:1261] T 21dd29f1917a47f7ab496136a4c7ca07 P a068ef6770f14874bd1acb01bce2c329 [term 3 FOLLOWER]: Refusing update from remote peer 3d7668f070e74286b38fda9d898f1acb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 20:00:09.885593 28550 catalog_manager.cc:5484] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb reported cstate change: term changed from 2 to 3, leader changed from dabdbfb59e8645c297a1696630198b6e (127.27.217.193) to 3d7668f070e74286b38fda9d898f1acb (127.27.217.195). New cstate: current_term: 3 leader_uuid: "3d7668f070e74286b38fda9d898f1acb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } health_report { overall_health: HEALTHY } } }
I20240501 20:00:09.887022 28943 consensus_queue.cc:1035] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 20:00:10.293761 28654 raft_consensus.cc:1261] T 6a175ef3d4c94d459208ea5cfc31e9d8 P dabdbfb59e8645c297a1696630198b6e [term 3 FOLLOWER]: Refusing update from remote peer 3d7668f070e74286b38fda9d898f1acb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 20:00:10.294909 28953 consensus_queue.cc:1035] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:00:10.302270 28728 raft_consensus.cc:1261] T 6a175ef3d4c94d459208ea5cfc31e9d8 P a068ef6770f14874bd1acb01bce2c329 [term 3 FOLLOWER]: Refusing update from remote peer 3d7668f070e74286b38fda9d898f1acb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 20:00:10.303478 28953 consensus_queue.cc:1035] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:00:10.457556 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:10.462532 28519 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:10.462967 28519 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:11.463800 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:11.469066 28519 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:11.469427 28519 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:12.470080 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:12.474879 28519 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:12.475237 28519 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:13.476361 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:13.481600 28519 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:13.481966 28519 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:14.482734 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:14.488469 28519 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:14.488968 28519 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:15.489818 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:15.495054 28519 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:15.495463 28519 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:16.496387 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:16.501041 28519 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:16.501435 28519 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:17.502305 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:17.508249 28519 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:17.508617 28519 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:18.509512 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:18.514312 28519 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:18.514689 28519 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:19.536132 28519 auto_leader_rebalancer-test.cc:280] The leader distribution is
I20240501 20:00:19.540524 28654 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2ddf397fabe8453b8e1601a863a1b84e"
dest_uuid: "dabdbfb59e8645c297a1696630198b6e"
mode: GRACEFUL
new_leader_uuid: "a068ef6770f14874bd1acb01bce2c329"
from {username='slave'} at 127.0.0.1:42564
I20240501 20:00:19.541149 28654 raft_consensus.cc:595] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e [term 1 LEADER]: Received request to transfer leadership to TS a068ef6770f14874bd1acb01bce2c329
I20240501 20:00:19.545428 28802 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6a175ef3d4c94d459208ea5cfc31e9d8"
dest_uuid: "3d7668f070e74286b38fda9d898f1acb"
mode: GRACEFUL
new_leader_uuid: "a068ef6770f14874bd1acb01bce2c329"
from {username='slave'} at 127.0.0.1:57390
I20240501 20:00:19.545835 28802 raft_consensus.cc:595] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb [term 3 LEADER]: Received request to transfer leadership to TS a068ef6770f14874bd1acb01bce2c329
I20240501 20:00:19.549870 28654 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8a0820e7c49f40f89038e0af73567bac"
dest_uuid: "dabdbfb59e8645c297a1696630198b6e"
mode: GRACEFUL
new_leader_uuid: "a068ef6770f14874bd1acb01bce2c329"
from {username='slave'} at 127.0.0.1:42564
I20240501 20:00:19.550256 28654 raft_consensus.cc:595] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e [term 2 LEADER]: Received request to transfer leadership to TS a068ef6770f14874bd1acb01bce2c329
I20240501 20:00:19.553942 28654 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "676f538543c2491399b66b7db1490445"
dest_uuid: "dabdbfb59e8645c297a1696630198b6e"
mode: GRACEFUL
new_leader_uuid: "a068ef6770f14874bd1acb01bce2c329"
from {username='slave'} at 127.0.0.1:42564
I20240501 20:00:19.554334 28654 raft_consensus.cc:595] T 676f538543c2491399b66b7db1490445 P dabdbfb59e8645c297a1696630198b6e [term 1 LEADER]: Received request to transfer leadership to TS a068ef6770f14874bd1acb01bce2c329
I20240501 20:00:19.558145 28802 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "21dd29f1917a47f7ab496136a4c7ca07"
dest_uuid: "3d7668f070e74286b38fda9d898f1acb"
mode: GRACEFUL
new_leader_uuid: "a068ef6770f14874bd1acb01bce2c329"
from {username='slave'} at 127.0.0.1:57390
I20240501 20:00:19.558532 28802 raft_consensus.cc:595] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb [term 3 LEADER]: Received request to transfer leadership to TS a068ef6770f14874bd1acb01bce2c329
I20240501 20:00:19.563212 28962 raft_consensus.cc:979] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb: : Instructing follower a068ef6770f14874bd1acb01bce2c329 to start an election
I20240501 20:00:19.563730 28967 raft_consensus.cc:1067] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb [term 3 LEADER]: Signalling peer a068ef6770f14874bd1acb01bce2c329 to start an election
I20240501 20:00:19.565171 28728 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "21dd29f1917a47f7ab496136a4c7ca07"
dest_uuid: "a068ef6770f14874bd1acb01bce2c329"
from {username='slave'} at 127.0.0.1:48486
I20240501 20:00:19.565685 28728 raft_consensus.cc:482] T 21dd29f1917a47f7ab496136a4c7ca07 P a068ef6770f14874bd1acb01bce2c329 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:00:19.566103 28728 raft_consensus.cc:3037] T 21dd29f1917a47f7ab496136a4c7ca07 P a068ef6770f14874bd1acb01bce2c329 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:00:19.571514 28728 raft_consensus.cc:504] T 21dd29f1917a47f7ab496136a4c7ca07 P a068ef6770f14874bd1acb01bce2c329 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:19.573040 28728 leader_election.cc:288] T 21dd29f1917a47f7ab496136a4c7ca07 P a068ef6770f14874bd1acb01bce2c329 [CANDIDATE]: Term 4 election: Requested vote from peers dabdbfb59e8645c297a1696630198b6e (127.27.217.193:46801), 3d7668f070e74286b38fda9d898f1acb (127.27.217.195:38723)
I20240501 20:00:19.573711 28654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21dd29f1917a47f7ab496136a4c7ca07" candidate_uuid: "a068ef6770f14874bd1acb01bce2c329" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "dabdbfb59e8645c297a1696630198b6e"
I20240501 20:00:19.574576 28654 raft_consensus.cc:3037] T 21dd29f1917a47f7ab496136a4c7ca07 P dabdbfb59e8645c297a1696630198b6e [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:00:19.574079 28802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21dd29f1917a47f7ab496136a4c7ca07" candidate_uuid: "a068ef6770f14874bd1acb01bce2c329" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "3d7668f070e74286b38fda9d898f1acb"
I20240501 20:00:19.576611 28802 raft_consensus.cc:3032] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:00:19.576926 28802 raft_consensus.cc:729] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3d7668f070e74286b38fda9d898f1acb, State: Running, Role: LEADER
I20240501 20:00:19.577661 28802 consensus_queue.cc:260] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:19.578910 28802 raft_consensus.cc:3037] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:00:19.579838 28654 raft_consensus.cc:2454] T 21dd29f1917a47f7ab496136a4c7ca07 P dabdbfb59e8645c297a1696630198b6e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a068ef6770f14874bd1acb01bce2c329 in term 4.
I20240501 20:00:19.581029 28694 leader_election.cc:302] T 21dd29f1917a47f7ab496136a4c7ca07 P a068ef6770f14874bd1acb01bce2c329 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a068ef6770f14874bd1acb01bce2c329, dabdbfb59e8645c297a1696630198b6e; no voters:
I20240501 20:00:19.581832 28965 raft_consensus.cc:2781] T 21dd29f1917a47f7ab496136a4c7ca07 P a068ef6770f14874bd1acb01bce2c329 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:00:19.582820 28965 raft_consensus.cc:686] T 21dd29f1917a47f7ab496136a4c7ca07 P a068ef6770f14874bd1acb01bce2c329 [term 4 LEADER]: Becoming Leader. State: Replica: a068ef6770f14874bd1acb01bce2c329, State: Running, Role: LEADER
I20240501 20:00:19.583554 28965 consensus_queue.cc:237] T 21dd29f1917a47f7ab496136a4c7ca07 P a068ef6770f14874bd1acb01bce2c329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:19.583974 28802 raft_consensus.cc:2454] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a068ef6770f14874bd1acb01bce2c329 in term 4.
I20240501 20:00:19.590538 28550 catalog_manager.cc:5484] T 21dd29f1917a47f7ab496136a4c7ca07 P a068ef6770f14874bd1acb01bce2c329 reported cstate change: term changed from 3 to 4, leader changed from 3d7668f070e74286b38fda9d898f1acb (127.27.217.195) to a068ef6770f14874bd1acb01bce2c329 (127.27.217.194). New cstate: current_term: 4 leader_uuid: "a068ef6770f14874bd1acb01bce2c329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } health_report { overall_health: UNKNOWN } } }
I20240501 20:00:19.643865 28962 raft_consensus.cc:979] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb: : Instructing follower a068ef6770f14874bd1acb01bce2c329 to start an election
I20240501 20:00:19.644297 28967 raft_consensus.cc:1067] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb [term 3 LEADER]: Signalling peer a068ef6770f14874bd1acb01bce2c329 to start an election
I20240501 20:00:19.645619 28728 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6a175ef3d4c94d459208ea5cfc31e9d8"
dest_uuid: "a068ef6770f14874bd1acb01bce2c329"
from {username='slave'} at 127.0.0.1:48486
I20240501 20:00:19.646032 28728 raft_consensus.cc:482] T 6a175ef3d4c94d459208ea5cfc31e9d8 P a068ef6770f14874bd1acb01bce2c329 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:00:19.646297 28728 raft_consensus.cc:3037] T 6a175ef3d4c94d459208ea5cfc31e9d8 P a068ef6770f14874bd1acb01bce2c329 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:00:19.650298 28728 raft_consensus.cc:504] T 6a175ef3d4c94d459208ea5cfc31e9d8 P a068ef6770f14874bd1acb01bce2c329 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:19.651701 28728 leader_election.cc:288] T 6a175ef3d4c94d459208ea5cfc31e9d8 P a068ef6770f14874bd1acb01bce2c329 [CANDIDATE]: Term 4 election: Requested vote from peers dabdbfb59e8645c297a1696630198b6e (127.27.217.193:46801), 3d7668f070e74286b38fda9d898f1acb (127.27.217.195:38723)
I20240501 20:00:19.652524 28654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a175ef3d4c94d459208ea5cfc31e9d8" candidate_uuid: "a068ef6770f14874bd1acb01bce2c329" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "dabdbfb59e8645c297a1696630198b6e"
I20240501 20:00:19.652676 28802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a175ef3d4c94d459208ea5cfc31e9d8" candidate_uuid: "a068ef6770f14874bd1acb01bce2c329" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "3d7668f070e74286b38fda9d898f1acb"
I20240501 20:00:19.652958 28654 raft_consensus.cc:3037] T 6a175ef3d4c94d459208ea5cfc31e9d8 P dabdbfb59e8645c297a1696630198b6e [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:00:19.653163 28802 raft_consensus.cc:3032] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:00:19.653437 28802 raft_consensus.cc:729] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3d7668f070e74286b38fda9d898f1acb, State: Running, Role: LEADER
I20240501 20:00:19.653954 28802 consensus_queue.cc:260] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:19.654759 28802 raft_consensus.cc:3037] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:00:19.656951 28654 raft_consensus.cc:2454] T 6a175ef3d4c94d459208ea5cfc31e9d8 P dabdbfb59e8645c297a1696630198b6e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a068ef6770f14874bd1acb01bce2c329 in term 4.
I20240501 20:00:19.657766 28694 leader_election.cc:302] T 6a175ef3d4c94d459208ea5cfc31e9d8 P a068ef6770f14874bd1acb01bce2c329 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a068ef6770f14874bd1acb01bce2c329, dabdbfb59e8645c297a1696630198b6e; no voters:
I20240501 20:00:19.658380 28965 raft_consensus.cc:2781] T 6a175ef3d4c94d459208ea5cfc31e9d8 P a068ef6770f14874bd1acb01bce2c329 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:00:19.658740 28965 raft_consensus.cc:686] T 6a175ef3d4c94d459208ea5cfc31e9d8 P a068ef6770f14874bd1acb01bce2c329 [term 4 LEADER]: Becoming Leader. State: Replica: a068ef6770f14874bd1acb01bce2c329, State: Running, Role: LEADER
I20240501 20:00:19.659083 28802 raft_consensus.cc:2454] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a068ef6770f14874bd1acb01bce2c329 in term 4.
I20240501 20:00:19.659502 28965 consensus_queue.cc:237] T 6a175ef3d4c94d459208ea5cfc31e9d8 P a068ef6770f14874bd1acb01bce2c329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:19.665719 28550 catalog_manager.cc:5484] T 6a175ef3d4c94d459208ea5cfc31e9d8 P a068ef6770f14874bd1acb01bce2c329 reported cstate change: term changed from 3 to 4, leader changed from 3d7668f070e74286b38fda9d898f1acb (127.27.217.195) to a068ef6770f14874bd1acb01bce2c329 (127.27.217.194). New cstate: current_term: 4 leader_uuid: "a068ef6770f14874bd1acb01bce2c329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } health_report { overall_health: UNKNOWN } } }
I20240501 20:00:19.713374 28964 raft_consensus.cc:979] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e: : Instructing follower a068ef6770f14874bd1acb01bce2c329 to start an election
I20240501 20:00:19.714015 28966 raft_consensus.cc:1067] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e [term 1 LEADER]: Signalling peer a068ef6770f14874bd1acb01bce2c329 to start an election
I20240501 20:00:19.715443 28728 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2ddf397fabe8453b8e1601a863a1b84e"
dest_uuid: "a068ef6770f14874bd1acb01bce2c329"
from {username='slave'} at 127.0.0.1:48472
I20240501 20:00:19.715843 28728 raft_consensus.cc:482] T 2ddf397fabe8453b8e1601a863a1b84e P a068ef6770f14874bd1acb01bce2c329 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:00:19.716115 28728 raft_consensus.cc:3037] T 2ddf397fabe8453b8e1601a863a1b84e P a068ef6770f14874bd1acb01bce2c329 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:19.720296 28728 raft_consensus.cc:504] T 2ddf397fabe8453b8e1601a863a1b84e P a068ef6770f14874bd1acb01bce2c329 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } }
I20240501 20:00:19.721635 28728 leader_election.cc:288] T 2ddf397fabe8453b8e1601a863a1b84e P a068ef6770f14874bd1acb01bce2c329 [CANDIDATE]: Term 2 election: Requested vote from peers 3d7668f070e74286b38fda9d898f1acb (127.27.217.195:38723), dabdbfb59e8645c297a1696630198b6e (127.27.217.193:46801)
I20240501 20:00:19.722359 28802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ddf397fabe8453b8e1601a863a1b84e" candidate_uuid: "a068ef6770f14874bd1acb01bce2c329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3d7668f070e74286b38fda9d898f1acb"
I20240501 20:00:19.722731 28654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ddf397fabe8453b8e1601a863a1b84e" candidate_uuid: "a068ef6770f14874bd1acb01bce2c329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dabdbfb59e8645c297a1696630198b6e"
I20240501 20:00:19.722903 28802 raft_consensus.cc:3037] T 2ddf397fabe8453b8e1601a863a1b84e P 3d7668f070e74286b38fda9d898f1acb [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:19.723197 28654 raft_consensus.cc:3032] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:00:19.723410 28654 raft_consensus.cc:729] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dabdbfb59e8645c297a1696630198b6e, State: Running, Role: LEADER
I20240501 20:00:19.723963 28654 consensus_queue.cc:260] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } }
I20240501 20:00:19.724848 28654 raft_consensus.cc:3037] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:19.726917 28802 raft_consensus.cc:2454] T 2ddf397fabe8453b8e1601a863a1b84e P 3d7668f070e74286b38fda9d898f1acb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a068ef6770f14874bd1acb01bce2c329 in term 2.
I20240501 20:00:19.728220 28692 leader_election.cc:302] T 2ddf397fabe8453b8e1601a863a1b84e P a068ef6770f14874bd1acb01bce2c329 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d7668f070e74286b38fda9d898f1acb, a068ef6770f14874bd1acb01bce2c329; no voters:
I20240501 20:00:19.728972 28965 raft_consensus.cc:2781] T 2ddf397fabe8453b8e1601a863a1b84e P a068ef6770f14874bd1acb01bce2c329 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:00:19.729436 28654 raft_consensus.cc:2454] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a068ef6770f14874bd1acb01bce2c329 in term 2.
I20240501 20:00:19.730382 28965 raft_consensus.cc:686] T 2ddf397fabe8453b8e1601a863a1b84e P a068ef6770f14874bd1acb01bce2c329 [term 2 LEADER]: Becoming Leader. State: Replica: a068ef6770f14874bd1acb01bce2c329, State: Running, Role: LEADER
I20240501 20:00:19.731071 28965 consensus_queue.cc:237] T 2ddf397fabe8453b8e1601a863a1b84e P a068ef6770f14874bd1acb01bce2c329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } }
I20240501 20:00:19.736697 28550 catalog_manager.cc:5484] T 2ddf397fabe8453b8e1601a863a1b84e P a068ef6770f14874bd1acb01bce2c329 reported cstate change: term changed from 1 to 2, leader changed from dabdbfb59e8645c297a1696630198b6e (127.27.217.193) to a068ef6770f14874bd1acb01bce2c329 (127.27.217.194). New cstate: current_term: 2 leader_uuid: "a068ef6770f14874bd1acb01bce2c329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } health_report { overall_health: UNKNOWN } } }
I20240501 20:00:19.819331 28964 raft_consensus.cc:979] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e: : Instructing follower a068ef6770f14874bd1acb01bce2c329 to start an election
I20240501 20:00:19.819725 28966 raft_consensus.cc:1067] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e [term 2 LEADER]: Signalling peer a068ef6770f14874bd1acb01bce2c329 to start an election
I20240501 20:00:19.820981 28728 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8a0820e7c49f40f89038e0af73567bac"
dest_uuid: "a068ef6770f14874bd1acb01bce2c329"
from {username='slave'} at 127.0.0.1:48472
I20240501 20:00:19.821383 28728 raft_consensus.cc:482] T 8a0820e7c49f40f89038e0af73567bac P a068ef6770f14874bd1acb01bce2c329 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:00:19.821612 28728 raft_consensus.cc:3037] T 8a0820e7c49f40f89038e0af73567bac P a068ef6770f14874bd1acb01bce2c329 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:00:19.825318 28728 raft_consensus.cc:504] T 8a0820e7c49f40f89038e0af73567bac P a068ef6770f14874bd1acb01bce2c329 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:19.826637 28728 leader_election.cc:288] T 8a0820e7c49f40f89038e0af73567bac P a068ef6770f14874bd1acb01bce2c329 [CANDIDATE]: Term 3 election: Requested vote from peers dabdbfb59e8645c297a1696630198b6e (127.27.217.193:46801), 3d7668f070e74286b38fda9d898f1acb (127.27.217.195:38723)
I20240501 20:00:19.827350 28654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0820e7c49f40f89038e0af73567bac" candidate_uuid: "a068ef6770f14874bd1acb01bce2c329" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "dabdbfb59e8645c297a1696630198b6e"
I20240501 20:00:19.827575 28802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0820e7c49f40f89038e0af73567bac" candidate_uuid: "a068ef6770f14874bd1acb01bce2c329" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3d7668f070e74286b38fda9d898f1acb"
I20240501 20:00:19.827796 28654 raft_consensus.cc:3032] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:00:19.828105 28654 raft_consensus.cc:729] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: dabdbfb59e8645c297a1696630198b6e, State: Running, Role: LEADER
I20240501 20:00:19.828135 28802 raft_consensus.cc:3037] T 8a0820e7c49f40f89038e0af73567bac P 3d7668f070e74286b38fda9d898f1acb [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:00:19.828742 28654 consensus_queue.cc:260] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:19.829602 28654 raft_consensus.cc:3037] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:00:19.832542 28802 raft_consensus.cc:2454] T 8a0820e7c49f40f89038e0af73567bac P 3d7668f070e74286b38fda9d898f1acb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a068ef6770f14874bd1acb01bce2c329 in term 3.
I20240501 20:00:19.833374 28654 raft_consensus.cc:2454] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a068ef6770f14874bd1acb01bce2c329 in term 3.
I20240501 20:00:19.833350 28692 leader_election.cc:302] T 8a0820e7c49f40f89038e0af73567bac P a068ef6770f14874bd1acb01bce2c329 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d7668f070e74286b38fda9d898f1acb, a068ef6770f14874bd1acb01bce2c329; no voters:
I20240501 20:00:19.834169 28965 raft_consensus.cc:2781] T 8a0820e7c49f40f89038e0af73567bac P a068ef6770f14874bd1acb01bce2c329 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:00:19.834522 28965 raft_consensus.cc:686] T 8a0820e7c49f40f89038e0af73567bac P a068ef6770f14874bd1acb01bce2c329 [term 3 LEADER]: Becoming Leader. State: Replica: a068ef6770f14874bd1acb01bce2c329, State: Running, Role: LEADER
I20240501 20:00:19.835254 28965 consensus_queue.cc:237] T 8a0820e7c49f40f89038e0af73567bac P a068ef6770f14874bd1acb01bce2c329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:19.840970 28550 catalog_manager.cc:5484] T 8a0820e7c49f40f89038e0af73567bac P a068ef6770f14874bd1acb01bce2c329 reported cstate change: term changed from 2 to 3, leader changed from dabdbfb59e8645c297a1696630198b6e (127.27.217.193) to a068ef6770f14874bd1acb01bce2c329 (127.27.217.194). New cstate: current_term: 3 leader_uuid: "a068ef6770f14874bd1acb01bce2c329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } health_report { overall_health: UNKNOWN } } }
I20240501 20:00:19.996387 28964 raft_consensus.cc:979] T 676f538543c2491399b66b7db1490445 P dabdbfb59e8645c297a1696630198b6e: : Instructing follower a068ef6770f14874bd1acb01bce2c329 to start an election
I20240501 20:00:19.996812 28964 raft_consensus.cc:1067] T 676f538543c2491399b66b7db1490445 P dabdbfb59e8645c297a1696630198b6e [term 1 LEADER]: Signalling peer a068ef6770f14874bd1acb01bce2c329 to start an election
I20240501 20:00:19.997977 28728 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "676f538543c2491399b66b7db1490445"
dest_uuid: "a068ef6770f14874bd1acb01bce2c329"
from {username='slave'} at 127.0.0.1:48472
I20240501 20:00:19.998382 28728 raft_consensus.cc:482] T 676f538543c2491399b66b7db1490445 P a068ef6770f14874bd1acb01bce2c329 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:00:19.998625 28728 raft_consensus.cc:3037] T 676f538543c2491399b66b7db1490445 P a068ef6770f14874bd1acb01bce2c329 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:20.002494 28728 raft_consensus.cc:504] T 676f538543c2491399b66b7db1490445 P a068ef6770f14874bd1acb01bce2c329 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:20.003870 28728 leader_election.cc:288] T 676f538543c2491399b66b7db1490445 P a068ef6770f14874bd1acb01bce2c329 [CANDIDATE]: Term 2 election: Requested vote from peers dabdbfb59e8645c297a1696630198b6e (127.27.217.193:46801), 3d7668f070e74286b38fda9d898f1acb (127.27.217.195:38723)
I20240501 20:00:20.004621 28654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "676f538543c2491399b66b7db1490445" candidate_uuid: "a068ef6770f14874bd1acb01bce2c329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dabdbfb59e8645c297a1696630198b6e"
I20240501 20:00:20.004873 28802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "676f538543c2491399b66b7db1490445" candidate_uuid: "a068ef6770f14874bd1acb01bce2c329" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3d7668f070e74286b38fda9d898f1acb"
I20240501 20:00:20.005192 28654 raft_consensus.cc:3032] T 676f538543c2491399b66b7db1490445 P dabdbfb59e8645c297a1696630198b6e [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:00:20.005378 28802 raft_consensus.cc:3037] T 676f538543c2491399b66b7db1490445 P 3d7668f070e74286b38fda9d898f1acb [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:20.005517 28654 raft_consensus.cc:729] T 676f538543c2491399b66b7db1490445 P dabdbfb59e8645c297a1696630198b6e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dabdbfb59e8645c297a1696630198b6e, State: Running, Role: LEADER
I20240501 20:00:20.006255 28654 consensus_queue.cc:260] T 676f538543c2491399b66b7db1490445 P dabdbfb59e8645c297a1696630198b6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:20.007371 28654 raft_consensus.cc:3037] T 676f538543c2491399b66b7db1490445 P dabdbfb59e8645c297a1696630198b6e [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:20.011324 28802 raft_consensus.cc:2454] T 676f538543c2491399b66b7db1490445 P 3d7668f070e74286b38fda9d898f1acb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a068ef6770f14874bd1acb01bce2c329 in term 2.
I20240501 20:00:20.012334 28692 leader_election.cc:302] T 676f538543c2491399b66b7db1490445 P a068ef6770f14874bd1acb01bce2c329 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d7668f070e74286b38fda9d898f1acb, a068ef6770f14874bd1acb01bce2c329; no voters:
I20240501 20:00:20.013185 28965 raft_consensus.cc:2781] T 676f538543c2491399b66b7db1490445 P a068ef6770f14874bd1acb01bce2c329 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:00:20.013188 28654 raft_consensus.cc:2454] T 676f538543c2491399b66b7db1490445 P dabdbfb59e8645c297a1696630198b6e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a068ef6770f14874bd1acb01bce2c329 in term 2.
I20240501 20:00:20.013703 28965 raft_consensus.cc:686] T 676f538543c2491399b66b7db1490445 P a068ef6770f14874bd1acb01bce2c329 [term 2 LEADER]: Becoming Leader. State: Replica: a068ef6770f14874bd1acb01bce2c329, State: Running, Role: LEADER
I20240501 20:00:20.014567 28965 consensus_queue.cc:237] T 676f538543c2491399b66b7db1490445 P a068ef6770f14874bd1acb01bce2c329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:20.020470 28550 catalog_manager.cc:5484] T 676f538543c2491399b66b7db1490445 P a068ef6770f14874bd1acb01bce2c329 reported cstate change: term changed from 1 to 2, leader changed from dabdbfb59e8645c297a1696630198b6e (127.27.217.193) to a068ef6770f14874bd1acb01bce2c329 (127.27.217.194). New cstate: current_term: 2 leader_uuid: "a068ef6770f14874bd1acb01bce2c329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } health_report { overall_health: UNKNOWN } } }
I20240501 20:00:20.094329 28802 raft_consensus.cc:1261] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb [term 4 FOLLOWER]: Refusing update from remote peer a068ef6770f14874bd1acb01bce2c329: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240501 20:00:20.095428 28965 consensus_queue.cc:1035] T 6a175ef3d4c94d459208ea5cfc31e9d8 P a068ef6770f14874bd1acb01bce2c329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:00:20.106096 28654 raft_consensus.cc:1261] T 21dd29f1917a47f7ab496136a4c7ca07 P dabdbfb59e8645c297a1696630198b6e [term 4 FOLLOWER]: Refusing update from remote peer a068ef6770f14874bd1acb01bce2c329: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240501 20:00:20.107581 28965 consensus_queue.cc:1035] T 21dd29f1917a47f7ab496136a4c7ca07 P a068ef6770f14874bd1acb01bce2c329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:00:20.108979 28654 raft_consensus.cc:1261] T 6a175ef3d4c94d459208ea5cfc31e9d8 P dabdbfb59e8645c297a1696630198b6e [term 4 FOLLOWER]: Refusing update from remote peer a068ef6770f14874bd1acb01bce2c329: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240501 20:00:20.110493 28975 consensus_queue.cc:1035] T 6a175ef3d4c94d459208ea5cfc31e9d8 P a068ef6770f14874bd1acb01bce2c329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240501 20:00:20.123272 28802 raft_consensus.cc:1261] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb [term 4 FOLLOWER]: Refusing update from remote peer a068ef6770f14874bd1acb01bce2c329: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240501 20:00:20.124696 28968 consensus_queue.cc:1035] T 21dd29f1917a47f7ab496136a4c7ca07 P a068ef6770f14874bd1acb01bce2c329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240501 20:00:20.273528 28802 raft_consensus.cc:1261] T 2ddf397fabe8453b8e1601a863a1b84e P 3d7668f070e74286b38fda9d898f1acb [term 2 FOLLOWER]: Refusing update from remote peer a068ef6770f14874bd1acb01bce2c329: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:00:20.274529 28968 consensus_queue.cc:1035] T 2ddf397fabe8453b8e1601a863a1b84e P a068ef6770f14874bd1acb01bce2c329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:00:20.282593 28654 raft_consensus.cc:1261] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e [term 2 FOLLOWER]: Refusing update from remote peer a068ef6770f14874bd1acb01bce2c329: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:00:20.283963 28965 consensus_queue.cc:1035] T 2ddf397fabe8453b8e1601a863a1b84e P a068ef6770f14874bd1acb01bce2c329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:00:20.309619 28654 raft_consensus.cc:1261] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e [term 3 FOLLOWER]: Refusing update from remote peer a068ef6770f14874bd1acb01bce2c329: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 20:00:20.310622 28965 consensus_queue.cc:1035] T 8a0820e7c49f40f89038e0af73567bac P a068ef6770f14874bd1acb01bce2c329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:00:20.320447 28802 raft_consensus.cc:1261] T 8a0820e7c49f40f89038e0af73567bac P 3d7668f070e74286b38fda9d898f1acb [term 3 FOLLOWER]: Refusing update from remote peer a068ef6770f14874bd1acb01bce2c329: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 20:00:20.321767 28965 consensus_queue.cc:1035] T 8a0820e7c49f40f89038e0af73567bac P a068ef6770f14874bd1acb01bce2c329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:00:20.398469 28802 raft_consensus.cc:1261] T 676f538543c2491399b66b7db1490445 P 3d7668f070e74286b38fda9d898f1acb [term 2 FOLLOWER]: Refusing update from remote peer a068ef6770f14874bd1acb01bce2c329: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:00:20.399500 28965 consensus_queue.cc:1035] T 676f538543c2491399b66b7db1490445 P a068ef6770f14874bd1acb01bce2c329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:00:20.408269 28654 raft_consensus.cc:1261] T 676f538543c2491399b66b7db1490445 P dabdbfb59e8645c297a1696630198b6e [term 2 FOLLOWER]: Refusing update from remote peer a068ef6770f14874bd1acb01bce2c329: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:00:20.409443 28968 consensus_queue.cc:1035] T 676f538543c2491399b66b7db1490445 P a068ef6770f14874bd1acb01bce2c329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:00:22.589821 28519 auto_leader_rebalancer-test.cc:295] The leader distribution is
I20240501 20:00:22.590369 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:22.597008 28728 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "101b282620ad41b28af298749f60ec52"
dest_uuid: "a068ef6770f14874bd1acb01bce2c329"
mode: GRACEFUL
new_leader_uuid: "3d7668f070e74286b38fda9d898f1acb"
from {username='slave'} at 127.0.0.1:48506
I20240501 20:00:22.597470 28728 raft_consensus.cc:595] T 101b282620ad41b28af298749f60ec52 P a068ef6770f14874bd1acb01bce2c329 [term 2 LEADER]: Received request to transfer leadership to TS 3d7668f070e74286b38fda9d898f1acb
I20240501 20:00:22.599499 28728 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2ddf397fabe8453b8e1601a863a1b84e"
dest_uuid: "a068ef6770f14874bd1acb01bce2c329"
mode: GRACEFUL
new_leader_uuid: "dabdbfb59e8645c297a1696630198b6e"
from {username='slave'} at 127.0.0.1:48506
I20240501 20:00:22.599910 28728 raft_consensus.cc:595] T 2ddf397fabe8453b8e1601a863a1b84e P a068ef6770f14874bd1acb01bce2c329 [term 2 LEADER]: Received request to transfer leadership to TS dabdbfb59e8645c297a1696630198b6e
I20240501 20:00:22.601732 28728 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8a0820e7c49f40f89038e0af73567bac"
dest_uuid: "a068ef6770f14874bd1acb01bce2c329"
mode: GRACEFUL
new_leader_uuid: "dabdbfb59e8645c297a1696630198b6e"
from {username='slave'} at 127.0.0.1:48506
I20240501 20:00:22.602125 28728 raft_consensus.cc:595] T 8a0820e7c49f40f89038e0af73567bac P a068ef6770f14874bd1acb01bce2c329 [term 3 LEADER]: Received request to transfer leadership to TS dabdbfb59e8645c297a1696630198b6e
I20240501 20:00:22.603827 28728 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8bd1afda996d441b8d3eaa64dd36d1c7"
dest_uuid: "a068ef6770f14874bd1acb01bce2c329"
mode: GRACEFUL
new_leader_uuid: "3d7668f070e74286b38fda9d898f1acb"
from {username='slave'} at 127.0.0.1:48506
I20240501 20:00:22.604221 28728 raft_consensus.cc:595] T 8bd1afda996d441b8d3eaa64dd36d1c7 P a068ef6770f14874bd1acb01bce2c329 [term 1 LEADER]: Received request to transfer leadership to TS 3d7668f070e74286b38fda9d898f1acb
I20240501 20:00:22.605971 28728 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fbcced7b903441278c70c4d3a1458be4"
dest_uuid: "a068ef6770f14874bd1acb01bce2c329"
mode: GRACEFUL
new_leader_uuid: "dabdbfb59e8645c297a1696630198b6e"
from {username='slave'} at 127.0.0.1:48506
I20240501 20:00:22.606348 28728 raft_consensus.cc:595] T fbcced7b903441278c70c4d3a1458be4 P a068ef6770f14874bd1acb01bce2c329 [term 1 LEADER]: Received request to transfer leadership to TS dabdbfb59e8645c297a1696630198b6e
I20240501 20:00:22.607141 28519 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240501 20:00:22.607537 28519 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:22.627336 28965 raft_consensus.cc:979] T 101b282620ad41b28af298749f60ec52 P a068ef6770f14874bd1acb01bce2c329: : Instructing follower 3d7668f070e74286b38fda9d898f1acb to start an election
I20240501 20:00:22.627787 28994 raft_consensus.cc:1067] T 101b282620ad41b28af298749f60ec52 P a068ef6770f14874bd1acb01bce2c329 [term 2 LEADER]: Signalling peer 3d7668f070e74286b38fda9d898f1acb to start an election
I20240501 20:00:22.629072 28802 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "101b282620ad41b28af298749f60ec52"
dest_uuid: "3d7668f070e74286b38fda9d898f1acb"
from {username='slave'} at 127.0.0.1:57378
I20240501 20:00:22.629477 28802 raft_consensus.cc:482] T 101b282620ad41b28af298749f60ec52 P 3d7668f070e74286b38fda9d898f1acb [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:00:22.629720 28802 raft_consensus.cc:3037] T 101b282620ad41b28af298749f60ec52 P 3d7668f070e74286b38fda9d898f1acb [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:00:22.633512 28802 raft_consensus.cc:504] T 101b282620ad41b28af298749f60ec52 P 3d7668f070e74286b38fda9d898f1acb [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:22.635186 28802 leader_election.cc:288] T 101b282620ad41b28af298749f60ec52 P 3d7668f070e74286b38fda9d898f1acb [CANDIDATE]: Term 3 election: Requested vote from peers dabdbfb59e8645c297a1696630198b6e (127.27.217.193:46801), a068ef6770f14874bd1acb01bce2c329 (127.27.217.194:37307)
I20240501 20:00:22.636078 28728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "101b282620ad41b28af298749f60ec52" candidate_uuid: "3d7668f070e74286b38fda9d898f1acb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a068ef6770f14874bd1acb01bce2c329"
I20240501 20:00:22.635999 28654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "101b282620ad41b28af298749f60ec52" candidate_uuid: "3d7668f070e74286b38fda9d898f1acb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "dabdbfb59e8645c297a1696630198b6e"
I20240501 20:00:22.636695 28728 raft_consensus.cc:3032] T 101b282620ad41b28af298749f60ec52 P a068ef6770f14874bd1acb01bce2c329 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:00:22.636765 28654 raft_consensus.cc:3037] T 101b282620ad41b28af298749f60ec52 P dabdbfb59e8645c297a1696630198b6e [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:00:22.636960 28728 raft_consensus.cc:729] T 101b282620ad41b28af298749f60ec52 P a068ef6770f14874bd1acb01bce2c329 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a068ef6770f14874bd1acb01bce2c329, State: Running, Role: LEADER
I20240501 20:00:22.637578 28728 consensus_queue.cc:260] T 101b282620ad41b28af298749f60ec52 P a068ef6770f14874bd1acb01bce2c329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:22.638378 28728 raft_consensus.cc:3037] T 101b282620ad41b28af298749f60ec52 P a068ef6770f14874bd1acb01bce2c329 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:00:22.640579 28654 raft_consensus.cc:2454] T 101b282620ad41b28af298749f60ec52 P dabdbfb59e8645c297a1696630198b6e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d7668f070e74286b38fda9d898f1acb in term 3.
I20240501 20:00:22.641489 28768 leader_election.cc:302] T 101b282620ad41b28af298749f60ec52 P 3d7668f070e74286b38fda9d898f1acb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d7668f070e74286b38fda9d898f1acb, dabdbfb59e8645c297a1696630198b6e; no voters:
I20240501 20:00:22.642105 28967 raft_consensus.cc:2781] T 101b282620ad41b28af298749f60ec52 P 3d7668f070e74286b38fda9d898f1acb [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:00:22.642314 28728 raft_consensus.cc:2454] T 101b282620ad41b28af298749f60ec52 P a068ef6770f14874bd1acb01bce2c329 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d7668f070e74286b38fda9d898f1acb in term 3.
I20240501 20:00:22.643049 28967 raft_consensus.cc:686] T 101b282620ad41b28af298749f60ec52 P 3d7668f070e74286b38fda9d898f1acb [term 3 LEADER]: Becoming Leader. State: Replica: 3d7668f070e74286b38fda9d898f1acb, State: Running, Role: LEADER
I20240501 20:00:22.643908 28967 consensus_queue.cc:237] T 101b282620ad41b28af298749f60ec52 P 3d7668f070e74286b38fda9d898f1acb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:22.650239 28550 catalog_manager.cc:5484] T 101b282620ad41b28af298749f60ec52 P 3d7668f070e74286b38fda9d898f1acb reported cstate change: term changed from 2 to 3, leader changed from a068ef6770f14874bd1acb01bce2c329 (127.27.217.194) to 3d7668f070e74286b38fda9d898f1acb (127.27.217.195). New cstate: current_term: 3 leader_uuid: "3d7668f070e74286b38fda9d898f1acb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } health_report { overall_health: HEALTHY } } }
I20240501 20:00:22.669335 28965 raft_consensus.cc:979] T fbcced7b903441278c70c4d3a1458be4 P a068ef6770f14874bd1acb01bce2c329: : Instructing follower dabdbfb59e8645c297a1696630198b6e to start an election
I20240501 20:00:22.669750 28994 raft_consensus.cc:1067] T fbcced7b903441278c70c4d3a1458be4 P a068ef6770f14874bd1acb01bce2c329 [term 1 LEADER]: Signalling peer dabdbfb59e8645c297a1696630198b6e to start an election
I20240501 20:00:22.670953 28654 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fbcced7b903441278c70c4d3a1458be4"
dest_uuid: "dabdbfb59e8645c297a1696630198b6e"
from {username='slave'} at 127.0.0.1:42542
I20240501 20:00:22.671473 28654 raft_consensus.cc:482] T fbcced7b903441278c70c4d3a1458be4 P dabdbfb59e8645c297a1696630198b6e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:00:22.671782 28654 raft_consensus.cc:3037] T fbcced7b903441278c70c4d3a1458be4 P dabdbfb59e8645c297a1696630198b6e [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:22.676239 28654 raft_consensus.cc:504] T fbcced7b903441278c70c4d3a1458be4 P dabdbfb59e8645c297a1696630198b6e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:22.677950 28654 leader_election.cc:288] T fbcced7b903441278c70c4d3a1458be4 P dabdbfb59e8645c297a1696630198b6e [CANDIDATE]: Term 2 election: Requested vote from peers a068ef6770f14874bd1acb01bce2c329 (127.27.217.194:37307), 3d7668f070e74286b38fda9d898f1acb (127.27.217.195:38723)
I20240501 20:00:22.678725 28728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbcced7b903441278c70c4d3a1458be4" candidate_uuid: "dabdbfb59e8645c297a1696630198b6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a068ef6770f14874bd1acb01bce2c329"
I20240501 20:00:22.678984 28802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbcced7b903441278c70c4d3a1458be4" candidate_uuid: "dabdbfb59e8645c297a1696630198b6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3d7668f070e74286b38fda9d898f1acb"
I20240501 20:00:22.679244 28728 raft_consensus.cc:3032] T fbcced7b903441278c70c4d3a1458be4 P a068ef6770f14874bd1acb01bce2c329 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:00:22.679559 28802 raft_consensus.cc:3037] T fbcced7b903441278c70c4d3a1458be4 P 3d7668f070e74286b38fda9d898f1acb [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:22.679574 28728 raft_consensus.cc:729] T fbcced7b903441278c70c4d3a1458be4 P a068ef6770f14874bd1acb01bce2c329 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a068ef6770f14874bd1acb01bce2c329, State: Running, Role: LEADER
I20240501 20:00:22.680455 28728 consensus_queue.cc:260] T fbcced7b903441278c70c4d3a1458be4 P a068ef6770f14874bd1acb01bce2c329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:22.681424 28728 raft_consensus.cc:3037] T fbcced7b903441278c70c4d3a1458be4 P a068ef6770f14874bd1acb01bce2c329 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:22.684021 28802 raft_consensus.cc:2454] T fbcced7b903441278c70c4d3a1458be4 P 3d7668f070e74286b38fda9d898f1acb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dabdbfb59e8645c297a1696630198b6e in term 2.
I20240501 20:00:22.684957 28617 leader_election.cc:302] T fbcced7b903441278c70c4d3a1458be4 P dabdbfb59e8645c297a1696630198b6e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d7668f070e74286b38fda9d898f1acb, dabdbfb59e8645c297a1696630198b6e; no voters:
I20240501 20:00:22.685200 28728 raft_consensus.cc:2454] T fbcced7b903441278c70c4d3a1458be4 P a068ef6770f14874bd1acb01bce2c329 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dabdbfb59e8645c297a1696630198b6e in term 2.
I20240501 20:00:22.686259 29001 raft_consensus.cc:2781] T fbcced7b903441278c70c4d3a1458be4 P dabdbfb59e8645c297a1696630198b6e [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:00:22.687213 29001 raft_consensus.cc:686] T fbcced7b903441278c70c4d3a1458be4 P dabdbfb59e8645c297a1696630198b6e [term 2 LEADER]: Becoming Leader. State: Replica: dabdbfb59e8645c297a1696630198b6e, State: Running, Role: LEADER
I20240501 20:00:22.687892 29001 consensus_queue.cc:237] T fbcced7b903441278c70c4d3a1458be4 P dabdbfb59e8645c297a1696630198b6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:22.693929 28550 catalog_manager.cc:5484] T fbcced7b903441278c70c4d3a1458be4 P dabdbfb59e8645c297a1696630198b6e reported cstate change: term changed from 1 to 2, leader changed from a068ef6770f14874bd1acb01bce2c329 (127.27.217.194) to dabdbfb59e8645c297a1696630198b6e (127.27.217.193). New cstate: current_term: 2 leader_uuid: "dabdbfb59e8645c297a1696630198b6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } health_report { overall_health: UNKNOWN } } }
I20240501 20:00:22.780217 28994 raft_consensus.cc:979] T 8a0820e7c49f40f89038e0af73567bac P a068ef6770f14874bd1acb01bce2c329: : Instructing follower dabdbfb59e8645c297a1696630198b6e to start an election
I20240501 20:00:22.780630 28965 raft_consensus.cc:1067] T 8a0820e7c49f40f89038e0af73567bac P a068ef6770f14874bd1acb01bce2c329 [term 3 LEADER]: Signalling peer dabdbfb59e8645c297a1696630198b6e to start an election
I20240501 20:00:22.781769 28654 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8a0820e7c49f40f89038e0af73567bac"
dest_uuid: "dabdbfb59e8645c297a1696630198b6e"
from {username='slave'} at 127.0.0.1:42542
I20240501 20:00:22.782145 28654 raft_consensus.cc:482] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:00:22.782354 28654 raft_consensus.cc:3037] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:00:22.786146 28654 raft_consensus.cc:504] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:22.787488 28654 leader_election.cc:288] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e [CANDIDATE]: Term 4 election: Requested vote from peers a068ef6770f14874bd1acb01bce2c329 (127.27.217.194:37307), 3d7668f070e74286b38fda9d898f1acb (127.27.217.195:38723)
I20240501 20:00:22.788143 28728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0820e7c49f40f89038e0af73567bac" candidate_uuid: "dabdbfb59e8645c297a1696630198b6e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "a068ef6770f14874bd1acb01bce2c329"
I20240501 20:00:22.788345 28802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a0820e7c49f40f89038e0af73567bac" candidate_uuid: "dabdbfb59e8645c297a1696630198b6e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "3d7668f070e74286b38fda9d898f1acb"
I20240501 20:00:22.788702 28728 raft_consensus.cc:3032] T 8a0820e7c49f40f89038e0af73567bac P a068ef6770f14874bd1acb01bce2c329 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:00:22.788791 28802 raft_consensus.cc:3037] T 8a0820e7c49f40f89038e0af73567bac P 3d7668f070e74286b38fda9d898f1acb [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:00:22.789042 28728 raft_consensus.cc:729] T 8a0820e7c49f40f89038e0af73567bac P a068ef6770f14874bd1acb01bce2c329 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a068ef6770f14874bd1acb01bce2c329, State: Running, Role: LEADER
I20240501 20:00:22.789731 28728 consensus_queue.cc:260] T 8a0820e7c49f40f89038e0af73567bac P a068ef6770f14874bd1acb01bce2c329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:22.790820 28728 raft_consensus.cc:3037] T 8a0820e7c49f40f89038e0af73567bac P a068ef6770f14874bd1acb01bce2c329 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:00:22.794412 28802 raft_consensus.cc:2454] T 8a0820e7c49f40f89038e0af73567bac P 3d7668f070e74286b38fda9d898f1acb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dabdbfb59e8645c297a1696630198b6e in term 4.
I20240501 20:00:22.795290 28617 leader_election.cc:302] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d7668f070e74286b38fda9d898f1acb, dabdbfb59e8645c297a1696630198b6e; no voters:
I20240501 20:00:22.795848 29001 raft_consensus.cc:2781] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:00:22.796115 28728 raft_consensus.cc:2454] T 8a0820e7c49f40f89038e0af73567bac P a068ef6770f14874bd1acb01bce2c329 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dabdbfb59e8645c297a1696630198b6e in term 4.
I20240501 20:00:22.796365 29001 raft_consensus.cc:686] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e [term 4 LEADER]: Becoming Leader. State: Replica: dabdbfb59e8645c297a1696630198b6e, State: Running, Role: LEADER
I20240501 20:00:22.797078 29001 consensus_queue.cc:237] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:22.803205 28550 catalog_manager.cc:5484] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e reported cstate change: term changed from 3 to 4, leader changed from a068ef6770f14874bd1acb01bce2c329 (127.27.217.194) to dabdbfb59e8645c297a1696630198b6e (127.27.217.193). New cstate: current_term: 4 leader_uuid: "dabdbfb59e8645c297a1696630198b6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } health_report { overall_health: UNKNOWN } } }
I20240501 20:00:22.887426 28994 raft_consensus.cc:979] T 2ddf397fabe8453b8e1601a863a1b84e P a068ef6770f14874bd1acb01bce2c329: : Instructing follower dabdbfb59e8645c297a1696630198b6e to start an election
I20240501 20:00:22.888096 28965 raft_consensus.cc:1067] T 2ddf397fabe8453b8e1601a863a1b84e P a068ef6770f14874bd1acb01bce2c329 [term 2 LEADER]: Signalling peer dabdbfb59e8645c297a1696630198b6e to start an election
I20240501 20:00:22.889600 28654 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2ddf397fabe8453b8e1601a863a1b84e"
dest_uuid: "dabdbfb59e8645c297a1696630198b6e"
from {username='slave'} at 127.0.0.1:42542
I20240501 20:00:22.890200 28654 raft_consensus.cc:482] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:00:22.890504 28654 raft_consensus.cc:3037] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:00:22.898128 28654 raft_consensus.cc:504] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } }
I20240501 20:00:22.899691 28654 leader_election.cc:288] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e [CANDIDATE]: Term 3 election: Requested vote from peers 3d7668f070e74286b38fda9d898f1acb (127.27.217.195:38723), a068ef6770f14874bd1acb01bce2c329 (127.27.217.194:37307)
I20240501 20:00:22.901250 28802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ddf397fabe8453b8e1601a863a1b84e" candidate_uuid: "dabdbfb59e8645c297a1696630198b6e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3d7668f070e74286b38fda9d898f1acb"
I20240501 20:00:22.901407 28728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ddf397fabe8453b8e1601a863a1b84e" candidate_uuid: "dabdbfb59e8645c297a1696630198b6e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a068ef6770f14874bd1acb01bce2c329"
I20240501 20:00:22.901952 28728 raft_consensus.cc:3032] T 2ddf397fabe8453b8e1601a863a1b84e P a068ef6770f14874bd1acb01bce2c329 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:00:22.901957 28802 raft_consensus.cc:3037] T 2ddf397fabe8453b8e1601a863a1b84e P 3d7668f070e74286b38fda9d898f1acb [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:00:22.902254 28728 raft_consensus.cc:729] T 2ddf397fabe8453b8e1601a863a1b84e P a068ef6770f14874bd1acb01bce2c329 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a068ef6770f14874bd1acb01bce2c329, State: Running, Role: LEADER
I20240501 20:00:22.902907 28728 consensus_queue.cc:260] T 2ddf397fabe8453b8e1601a863a1b84e P a068ef6770f14874bd1acb01bce2c329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } }
I20240501 20:00:22.903913 28728 raft_consensus.cc:3037] T 2ddf397fabe8453b8e1601a863a1b84e P a068ef6770f14874bd1acb01bce2c329 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:00:22.906651 28802 raft_consensus.cc:2454] T 2ddf397fabe8453b8e1601a863a1b84e P 3d7668f070e74286b38fda9d898f1acb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dabdbfb59e8645c297a1696630198b6e in term 3.
I20240501 20:00:22.907673 28617 leader_election.cc:302] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d7668f070e74286b38fda9d898f1acb, dabdbfb59e8645c297a1696630198b6e; no voters:
I20240501 20:00:22.908668 29001 raft_consensus.cc:2781] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:00:22.908824 28728 raft_consensus.cc:2454] T 2ddf397fabe8453b8e1601a863a1b84e P a068ef6770f14874bd1acb01bce2c329 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dabdbfb59e8645c297a1696630198b6e in term 3.
I20240501 20:00:22.909062 29001 raft_consensus.cc:686] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e [term 3 LEADER]: Becoming Leader. State: Replica: dabdbfb59e8645c297a1696630198b6e, State: Running, Role: LEADER
I20240501 20:00:22.910159 29001 consensus_queue.cc:237] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } }
I20240501 20:00:22.918042 28550 catalog_manager.cc:5484] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e reported cstate change: term changed from 2 to 3, leader changed from a068ef6770f14874bd1acb01bce2c329 (127.27.217.194) to dabdbfb59e8645c297a1696630198b6e (127.27.217.193). New cstate: current_term: 3 leader_uuid: "dabdbfb59e8645c297a1696630198b6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } health_report { overall_health: HEALTHY } } }
I20240501 20:00:23.066054 28728 raft_consensus.cc:1261] T 101b282620ad41b28af298749f60ec52 P a068ef6770f14874bd1acb01bce2c329 [term 3 FOLLOWER]: Refusing update from remote peer 3d7668f070e74286b38fda9d898f1acb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 20:00:23.067224 28967 consensus_queue.cc:1035] T 101b282620ad41b28af298749f60ec52 P 3d7668f070e74286b38fda9d898f1acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:00:23.077261 28654 raft_consensus.cc:1261] T 101b282620ad41b28af298749f60ec52 P dabdbfb59e8645c297a1696630198b6e [term 3 FOLLOWER]: Refusing update from remote peer 3d7668f070e74286b38fda9d898f1acb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 20:00:23.079875 29009 consensus_queue.cc:1035] T 101b282620ad41b28af298749f60ec52 P 3d7668f070e74286b38fda9d898f1acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 20:00:23.092772 28802 raft_consensus.cc:1261] T fbcced7b903441278c70c4d3a1458be4 P 3d7668f070e74286b38fda9d898f1acb [term 2 FOLLOWER]: Refusing update from remote peer dabdbfb59e8645c297a1696630198b6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:00:23.093863 29001 consensus_queue.cc:1035] T fbcced7b903441278c70c4d3a1458be4 P dabdbfb59e8645c297a1696630198b6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:00:23.105968 28728 raft_consensus.cc:1261] T fbcced7b903441278c70c4d3a1458be4 P a068ef6770f14874bd1acb01bce2c329 [term 2 FOLLOWER]: Refusing update from remote peer dabdbfb59e8645c297a1696630198b6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:00:23.107431 29001 consensus_queue.cc:1035] T fbcced7b903441278c70c4d3a1458be4 P dabdbfb59e8645c297a1696630198b6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:00:23.166160 28965 raft_consensus.cc:979] T 8bd1afda996d441b8d3eaa64dd36d1c7 P a068ef6770f14874bd1acb01bce2c329: : Instructing follower 3d7668f070e74286b38fda9d898f1acb to start an election
I20240501 20:00:23.166584 28994 raft_consensus.cc:1067] T 8bd1afda996d441b8d3eaa64dd36d1c7 P a068ef6770f14874bd1acb01bce2c329 [term 1 LEADER]: Signalling peer 3d7668f070e74286b38fda9d898f1acb to start an election
I20240501 20:00:23.167881 28802 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8bd1afda996d441b8d3eaa64dd36d1c7"
dest_uuid: "3d7668f070e74286b38fda9d898f1acb"
from {username='slave'} at 127.0.0.1:57378
I20240501 20:00:23.168309 28802 raft_consensus.cc:482] T 8bd1afda996d441b8d3eaa64dd36d1c7 P 3d7668f070e74286b38fda9d898f1acb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:00:23.168551 28802 raft_consensus.cc:3037] T 8bd1afda996d441b8d3eaa64dd36d1c7 P 3d7668f070e74286b38fda9d898f1acb [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:23.172308 28802 raft_consensus.cc:504] T 8bd1afda996d441b8d3eaa64dd36d1c7 P 3d7668f070e74286b38fda9d898f1acb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:23.173583 28802 leader_election.cc:288] T 8bd1afda996d441b8d3eaa64dd36d1c7 P 3d7668f070e74286b38fda9d898f1acb [CANDIDATE]: Term 2 election: Requested vote from peers dabdbfb59e8645c297a1696630198b6e (127.27.217.193:46801), a068ef6770f14874bd1acb01bce2c329 (127.27.217.194:37307)
I20240501 20:00:23.174275 28654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd1afda996d441b8d3eaa64dd36d1c7" candidate_uuid: "3d7668f070e74286b38fda9d898f1acb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dabdbfb59e8645c297a1696630198b6e"
I20240501 20:00:23.174444 28728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd1afda996d441b8d3eaa64dd36d1c7" candidate_uuid: "3d7668f070e74286b38fda9d898f1acb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a068ef6770f14874bd1acb01bce2c329"
I20240501 20:00:23.174777 28654 raft_consensus.cc:3037] T 8bd1afda996d441b8d3eaa64dd36d1c7 P dabdbfb59e8645c297a1696630198b6e [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:23.174968 28728 raft_consensus.cc:3032] T 8bd1afda996d441b8d3eaa64dd36d1c7 P a068ef6770f14874bd1acb01bce2c329 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:00:23.175235 28728 raft_consensus.cc:729] T 8bd1afda996d441b8d3eaa64dd36d1c7 P a068ef6770f14874bd1acb01bce2c329 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a068ef6770f14874bd1acb01bce2c329, State: Running, Role: LEADER
I20240501 20:00:23.175829 28728 consensus_queue.cc:260] T 8bd1afda996d441b8d3eaa64dd36d1c7 P a068ef6770f14874bd1acb01bce2c329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:23.176693 28728 raft_consensus.cc:3037] T 8bd1afda996d441b8d3eaa64dd36d1c7 P a068ef6770f14874bd1acb01bce2c329 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:23.179160 28654 raft_consensus.cc:2454] T 8bd1afda996d441b8d3eaa64dd36d1c7 P dabdbfb59e8645c297a1696630198b6e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d7668f070e74286b38fda9d898f1acb in term 2.
I20240501 20:00:23.180178 28768 leader_election.cc:302] T 8bd1afda996d441b8d3eaa64dd36d1c7 P 3d7668f070e74286b38fda9d898f1acb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d7668f070e74286b38fda9d898f1acb, dabdbfb59e8645c297a1696630198b6e; no voters:
I20240501 20:00:23.180747 29009 raft_consensus.cc:2781] T 8bd1afda996d441b8d3eaa64dd36d1c7 P 3d7668f070e74286b38fda9d898f1acb [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:00:23.181177 29009 raft_consensus.cc:686] T 8bd1afda996d441b8d3eaa64dd36d1c7 P 3d7668f070e74286b38fda9d898f1acb [term 2 LEADER]: Becoming Leader. State: Replica: 3d7668f070e74286b38fda9d898f1acb, State: Running, Role: LEADER
I20240501 20:00:23.181754 28802 raft_consensus.cc:1261] T 8a0820e7c49f40f89038e0af73567bac P 3d7668f070e74286b38fda9d898f1acb [term 4 FOLLOWER]: Refusing update from remote peer dabdbfb59e8645c297a1696630198b6e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240501 20:00:23.182097 29009 consensus_queue.cc:237] T 8bd1afda996d441b8d3eaa64dd36d1c7 P 3d7668f070e74286b38fda9d898f1acb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } }
I20240501 20:00:23.182523 28728 raft_consensus.cc:2454] T 8bd1afda996d441b8d3eaa64dd36d1c7 P a068ef6770f14874bd1acb01bce2c329 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d7668f070e74286b38fda9d898f1acb in term 2.
I20240501 20:00:23.183221 29001 consensus_queue.cc:1035] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:00:23.190196 28548 catalog_manager.cc:5484] T 8bd1afda996d441b8d3eaa64dd36d1c7 P 3d7668f070e74286b38fda9d898f1acb reported cstate change: term changed from 1 to 2, leader changed from a068ef6770f14874bd1acb01bce2c329 (127.27.217.194) to 3d7668f070e74286b38fda9d898f1acb (127.27.217.195). New cstate: current_term: 2 leader_uuid: "3d7668f070e74286b38fda9d898f1acb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 } health_report { overall_health: HEALTHY } } }
I20240501 20:00:23.192456 28728 raft_consensus.cc:1261] T 8a0820e7c49f40f89038e0af73567bac P a068ef6770f14874bd1acb01bce2c329 [term 4 FOLLOWER]: Refusing update from remote peer dabdbfb59e8645c297a1696630198b6e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240501 20:00:23.194011 29001 consensus_queue.cc:1035] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:00:23.436173 28728 raft_consensus.cc:1261] T 2ddf397fabe8453b8e1601a863a1b84e P a068ef6770f14874bd1acb01bce2c329 [term 3 FOLLOWER]: Refusing update from remote peer dabdbfb59e8645c297a1696630198b6e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 20:00:23.437372 29017 consensus_queue.cc:1035] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:00:23.445168 28802 raft_consensus.cc:1261] T 2ddf397fabe8453b8e1601a863a1b84e P 3d7668f070e74286b38fda9d898f1acb [term 3 FOLLOWER]: Refusing update from remote peer dabdbfb59e8645c297a1696630198b6e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 20:00:23.446645 29001 consensus_queue.cc:1035] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d7668f070e74286b38fda9d898f1acb" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 38723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:00:23.608407 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:23.616206 28519 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:23.616590 28519 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:23.625923 28654 raft_consensus.cc:1261] T 8bd1afda996d441b8d3eaa64dd36d1c7 P dabdbfb59e8645c297a1696630198b6e [term 2 FOLLOWER]: Refusing update from remote peer 3d7668f070e74286b38fda9d898f1acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:00:23.627069 28995 consensus_queue.cc:1035] T 8bd1afda996d441b8d3eaa64dd36d1c7 P 3d7668f070e74286b38fda9d898f1acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabdbfb59e8645c297a1696630198b6e" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:00:23.634799 28728 raft_consensus.cc:1261] T 8bd1afda996d441b8d3eaa64dd36d1c7 P a068ef6770f14874bd1acb01bce2c329 [term 2 FOLLOWER]: Refusing update from remote peer 3d7668f070e74286b38fda9d898f1acb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:00:23.636368 28995 consensus_queue.cc:1035] T 8bd1afda996d441b8d3eaa64dd36d1c7 P 3d7668f070e74286b38fda9d898f1acb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a068ef6770f14874bd1acb01bce2c329" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 37307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:00:24.617265 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:24.624155 28519 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:24.624560 28519 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:25.625259 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:25.629916 28519 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:25.630337 28519 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:26.631156 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:26.635965 28519 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:26.636412 28519 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:27.637131 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:27.642028 28519 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:27.642442 28519 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:28.643191 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:28.647842 28519 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:28.648272 28519 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:29.649092 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:29.653827 28519 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:29.654238 28519 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:30.654937 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:30.659750 28519 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:30.660174 28519 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:31.660975 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:31.665606 28519 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:31.666028 28519 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:32.687672 28519 auto_leader_rebalancer-test.cc:306] The leader distribution is
I20240501 20:00:32.688279 28519 tablet_server.cc:178] TabletServer@127.27.217.193:0 shutting down...
W20240501 20:00:32.703069 28768 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 20:00:32.707793 28768 consensus_peers.cc:485] T 14bfbbeb889d43218d5807488de4fed2 P 3d7668f070e74286b38fda9d898f1acb -> Peer dabdbfb59e8645c297a1696630198b6e (127.27.217.193:46801): Couldn't send request to peer dabdbfb59e8645c297a1696630198b6e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:00:32.724318 28519 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:00:32.724969 28519 tablet_replica.cc:330] T 14bfbbeb889d43218d5807488de4fed2 P dabdbfb59e8645c297a1696630198b6e: stopping tablet replica
I20240501 20:00:32.725690 28519 raft_consensus.cc:2229] T 14bfbbeb889d43218d5807488de4fed2 P dabdbfb59e8645c297a1696630198b6e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:00:32.726208 28519 raft_consensus.cc:2258] T 14bfbbeb889d43218d5807488de4fed2 P dabdbfb59e8645c297a1696630198b6e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:32.728457 28519 tablet_replica.cc:330] T fbcced7b903441278c70c4d3a1458be4 P dabdbfb59e8645c297a1696630198b6e: stopping tablet replica
I20240501 20:00:32.728924 28519 raft_consensus.cc:2229] T fbcced7b903441278c70c4d3a1458be4 P dabdbfb59e8645c297a1696630198b6e [term 2 LEADER]: Raft consensus shutting down.
I20240501 20:00:32.729588 28519 raft_consensus.cc:2258] T fbcced7b903441278c70c4d3a1458be4 P dabdbfb59e8645c297a1696630198b6e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:32.731012 28519 tablet_replica.cc:330] T 676f538543c2491399b66b7db1490445 P dabdbfb59e8645c297a1696630198b6e: stopping tablet replica
I20240501 20:00:32.731510 28519 raft_consensus.cc:2229] T 676f538543c2491399b66b7db1490445 P dabdbfb59e8645c297a1696630198b6e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:00:32.731879 28519 raft_consensus.cc:2258] T 676f538543c2491399b66b7db1490445 P dabdbfb59e8645c297a1696630198b6e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:32.733353 28519 tablet_replica.cc:330] T 6a175ef3d4c94d459208ea5cfc31e9d8 P dabdbfb59e8645c297a1696630198b6e: stopping tablet replica
I20240501 20:00:32.733811 28519 raft_consensus.cc:2229] T 6a175ef3d4c94d459208ea5cfc31e9d8 P dabdbfb59e8645c297a1696630198b6e [term 4 FOLLOWER]: Raft consensus shutting down.
I20240501 20:00:32.734180 28519 raft_consensus.cc:2258] T 6a175ef3d4c94d459208ea5cfc31e9d8 P dabdbfb59e8645c297a1696630198b6e [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:32.735504 28519 tablet_replica.cc:330] T 101b282620ad41b28af298749f60ec52 P dabdbfb59e8645c297a1696630198b6e: stopping tablet replica
I20240501 20:00:32.735965 28519 raft_consensus.cc:2229] T 101b282620ad41b28af298749f60ec52 P dabdbfb59e8645c297a1696630198b6e [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 20:00:32.736389 28519 raft_consensus.cc:2258] T 101b282620ad41b28af298749f60ec52 P dabdbfb59e8645c297a1696630198b6e [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:32.737752 28519 tablet_replica.cc:330] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e: stopping tablet replica
I20240501 20:00:32.738188 28519 raft_consensus.cc:2229] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e [term 3 LEADER]: Raft consensus shutting down.
I20240501 20:00:32.738816 28519 raft_consensus.cc:2258] T 2ddf397fabe8453b8e1601a863a1b84e P dabdbfb59e8645c297a1696630198b6e [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:32.740159 28519 tablet_replica.cc:330] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e: stopping tablet replica
I20240501 20:00:32.740576 28519 raft_consensus.cc:2229] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e [term 4 LEADER]: Raft consensus shutting down.
I20240501 20:00:32.741199 28519 raft_consensus.cc:2258] T 8a0820e7c49f40f89038e0af73567bac P dabdbfb59e8645c297a1696630198b6e [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:32.742743 28519 tablet_replica.cc:330] T 21dd29f1917a47f7ab496136a4c7ca07 P dabdbfb59e8645c297a1696630198b6e: stopping tablet replica
I20240501 20:00:32.743173 28519 raft_consensus.cc:2229] T 21dd29f1917a47f7ab496136a4c7ca07 P dabdbfb59e8645c297a1696630198b6e [term 4 FOLLOWER]: Raft consensus shutting down.
I20240501 20:00:32.743547 28519 raft_consensus.cc:2258] T 21dd29f1917a47f7ab496136a4c7ca07 P dabdbfb59e8645c297a1696630198b6e [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:32.744997 28519 tablet_replica.cc:330] T 8bd1afda996d441b8d3eaa64dd36d1c7 P dabdbfb59e8645c297a1696630198b6e: stopping tablet replica
I20240501 20:00:32.745404 28519 raft_consensus.cc:2229] T 8bd1afda996d441b8d3eaa64dd36d1c7 P dabdbfb59e8645c297a1696630198b6e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:00:32.745728 28519 raft_consensus.cc:2258] T 8bd1afda996d441b8d3eaa64dd36d1c7 P dabdbfb59e8645c297a1696630198b6e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:32.853541 28519 tablet_server.cc:194] TabletServer@127.27.217.193:0 shutdown complete.
W20240501 20:00:32.853837 28768 consensus_peers.cc:485] T 8bd1afda996d441b8d3eaa64dd36d1c7 P 3d7668f070e74286b38fda9d898f1acb -> Peer dabdbfb59e8645c297a1696630198b6e (127.27.217.193:46801): Couldn't send request to peer dabdbfb59e8645c297a1696630198b6e. Status: Network error: Client connection negotiation failed: client connection to 127.27.217.193:46801: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:00:32.870330 28519 tablet_server.cc:178] TabletServer@127.27.217.194:0 shutting down...
W20240501 20:00:32.874980 28766 consensus_peers.cc:485] T 8bd1afda996d441b8d3eaa64dd36d1c7 P 3d7668f070e74286b38fda9d898f1acb -> Peer a068ef6770f14874bd1acb01bce2c329 (127.27.217.194:37307): Couldn't send request to peer a068ef6770f14874bd1acb01bce2c329. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20240501 20:00:32.905916 28766 consensus_peers.cc:485] T 101b282620ad41b28af298749f60ec52 P 3d7668f070e74286b38fda9d898f1acb -> Peer a068ef6770f14874bd1acb01bce2c329 (127.27.217.194:37307): Couldn't send request to peer a068ef6770f14874bd1acb01bce2c329. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:00:32.914213 28519 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:00:32.915102 28519 tablet_replica.cc:330] T 8bd1afda996d441b8d3eaa64dd36d1c7 P a068ef6770f14874bd1acb01bce2c329: stopping tablet replica
I20240501 20:00:32.916006 28519 raft_consensus.cc:2229] T 8bd1afda996d441b8d3eaa64dd36d1c7 P a068ef6770f14874bd1acb01bce2c329 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:00:32.916718 28519 raft_consensus.cc:2258] T 8bd1afda996d441b8d3eaa64dd36d1c7 P a068ef6770f14874bd1acb01bce2c329 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:32.918629 28519 tablet_replica.cc:330] T 6a175ef3d4c94d459208ea5cfc31e9d8 P a068ef6770f14874bd1acb01bce2c329: stopping tablet replica
I20240501 20:00:32.919239 28519 raft_consensus.cc:2229] T 6a175ef3d4c94d459208ea5cfc31e9d8 P a068ef6770f14874bd1acb01bce2c329 [term 4 LEADER]: Raft consensus shutting down.
I20240501 20:00:32.920462 28519 raft_consensus.cc:2258] T 6a175ef3d4c94d459208ea5cfc31e9d8 P a068ef6770f14874bd1acb01bce2c329 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:32.922204 28519 tablet_replica.cc:330] T 676f538543c2491399b66b7db1490445 P a068ef6770f14874bd1acb01bce2c329: stopping tablet replica
I20240501 20:00:32.922660 28519 raft_consensus.cc:2229] T 676f538543c2491399b66b7db1490445 P a068ef6770f14874bd1acb01bce2c329 [term 2 LEADER]: Raft consensus shutting down.
I20240501 20:00:32.923703 28519 raft_consensus.cc:2258] T 676f538543c2491399b66b7db1490445 P a068ef6770f14874bd1acb01bce2c329 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:32.925343 28519 tablet_replica.cc:330] T fbcced7b903441278c70c4d3a1458be4 P a068ef6770f14874bd1acb01bce2c329: stopping tablet replica
I20240501 20:00:32.925801 28519 raft_consensus.cc:2229] T fbcced7b903441278c70c4d3a1458be4 P a068ef6770f14874bd1acb01bce2c329 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:00:32.926262 28519 raft_consensus.cc:2258] T fbcced7b903441278c70c4d3a1458be4 P a068ef6770f14874bd1acb01bce2c329 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:32.927882 28519 tablet_replica.cc:330] T 8a0820e7c49f40f89038e0af73567bac P a068ef6770f14874bd1acb01bce2c329: stopping tablet replica
I20240501 20:00:32.928421 28519 raft_consensus.cc:2229] T 8a0820e7c49f40f89038e0af73567bac P a068ef6770f14874bd1acb01bce2c329 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240501 20:00:32.928911 28519 raft_consensus.cc:2258] T 8a0820e7c49f40f89038e0af73567bac P a068ef6770f14874bd1acb01bce2c329 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:32.930473 28519 tablet_replica.cc:330] T 21dd29f1917a47f7ab496136a4c7ca07 P a068ef6770f14874bd1acb01bce2c329: stopping tablet replica
I20240501 20:00:32.930965 28519 raft_consensus.cc:2229] T 21dd29f1917a47f7ab496136a4c7ca07 P a068ef6770f14874bd1acb01bce2c329 [term 4 LEADER]: Raft consensus shutting down.
I20240501 20:00:32.931886 28519 raft_consensus.cc:2258] T 21dd29f1917a47f7ab496136a4c7ca07 P a068ef6770f14874bd1acb01bce2c329 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:32.933485 28519 tablet_replica.cc:330] T 14bfbbeb889d43218d5807488de4fed2 P a068ef6770f14874bd1acb01bce2c329: stopping tablet replica
I20240501 20:00:32.933954 28519 raft_consensus.cc:2229] T 14bfbbeb889d43218d5807488de4fed2 P a068ef6770f14874bd1acb01bce2c329 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:00:32.934376 28519 raft_consensus.cc:2258] T 14bfbbeb889d43218d5807488de4fed2 P a068ef6770f14874bd1acb01bce2c329 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:32.935870 28519 tablet_replica.cc:330] T 101b282620ad41b28af298749f60ec52 P a068ef6770f14874bd1acb01bce2c329: stopping tablet replica
I20240501 20:00:32.936331 28519 raft_consensus.cc:2229] T 101b282620ad41b28af298749f60ec52 P a068ef6770f14874bd1acb01bce2c329 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 20:00:32.936750 28519 raft_consensus.cc:2258] T 101b282620ad41b28af298749f60ec52 P a068ef6770f14874bd1acb01bce2c329 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:32.938314 28519 tablet_replica.cc:330] T 2ddf397fabe8453b8e1601a863a1b84e P a068ef6770f14874bd1acb01bce2c329: stopping tablet replica
I20240501 20:00:32.938715 28519 raft_consensus.cc:2229] T 2ddf397fabe8453b8e1601a863a1b84e P a068ef6770f14874bd1acb01bce2c329 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 20:00:32.939138 28519 raft_consensus.cc:2258] T 2ddf397fabe8453b8e1601a863a1b84e P a068ef6770f14874bd1acb01bce2c329 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:32.967491 28519 tablet_server.cc:194] TabletServer@127.27.217.194:0 shutdown complete.
I20240501 20:00:32.984354 28519 tablet_server.cc:178] TabletServer@127.27.217.195:0 shutting down...
I20240501 20:00:33.014127 28519 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:00:33.014899 28519 tablet_replica.cc:330] T 676f538543c2491399b66b7db1490445 P 3d7668f070e74286b38fda9d898f1acb: stopping tablet replica
I20240501 20:00:33.015516 28519 raft_consensus.cc:2229] T 676f538543c2491399b66b7db1490445 P 3d7668f070e74286b38fda9d898f1acb [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:00:33.016000 28519 raft_consensus.cc:2258] T 676f538543c2491399b66b7db1490445 P 3d7668f070e74286b38fda9d898f1acb [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:33.018086 28519 tablet_replica.cc:330] T fbcced7b903441278c70c4d3a1458be4 P 3d7668f070e74286b38fda9d898f1acb: stopping tablet replica
I20240501 20:00:33.018682 28519 raft_consensus.cc:2229] T fbcced7b903441278c70c4d3a1458be4 P 3d7668f070e74286b38fda9d898f1acb [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:00:33.019282 28519 raft_consensus.cc:2258] T fbcced7b903441278c70c4d3a1458be4 P 3d7668f070e74286b38fda9d898f1acb [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:33.021355 28519 tablet_replica.cc:330] T 8bd1afda996d441b8d3eaa64dd36d1c7 P 3d7668f070e74286b38fda9d898f1acb: stopping tablet replica
I20240501 20:00:33.021828 28519 raft_consensus.cc:2229] T 8bd1afda996d441b8d3eaa64dd36d1c7 P 3d7668f070e74286b38fda9d898f1acb [term 2 LEADER]: Raft consensus shutting down.
W20240501 20:00:33.022434 28768 consensus_peers.cc:485] T 101b282620ad41b28af298749f60ec52 P 3d7668f070e74286b38fda9d898f1acb -> Peer dabdbfb59e8645c297a1696630198b6e (127.27.217.193:46801): Couldn't send request to peer dabdbfb59e8645c297a1696630198b6e. Status: Network error: Client connection negotiation failed: client connection to 127.27.217.193:46801: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:00:33.023079 28519 raft_consensus.cc:2258] T 8bd1afda996d441b8d3eaa64dd36d1c7 P 3d7668f070e74286b38fda9d898f1acb [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:33.024766 28519 tablet_replica.cc:330] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb: stopping tablet replica
I20240501 20:00:33.025229 28519 raft_consensus.cc:2229] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb [term 4 FOLLOWER]: Raft consensus shutting down.
I20240501 20:00:33.025656 28519 raft_consensus.cc:2258] T 6a175ef3d4c94d459208ea5cfc31e9d8 P 3d7668f070e74286b38fda9d898f1acb [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:33.027174 28519 tablet_replica.cc:330] T 101b282620ad41b28af298749f60ec52 P 3d7668f070e74286b38fda9d898f1acb: stopping tablet replica
I20240501 20:00:33.027626 28519 raft_consensus.cc:2229] T 101b282620ad41b28af298749f60ec52 P 3d7668f070e74286b38fda9d898f1acb [term 3 LEADER]: Raft consensus shutting down.
I20240501 20:00:33.028456 28519 raft_consensus.cc:2258] T 101b282620ad41b28af298749f60ec52 P 3d7668f070e74286b38fda9d898f1acb [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:33.030053 28519 tablet_replica.cc:330] T 2ddf397fabe8453b8e1601a863a1b84e P 3d7668f070e74286b38fda9d898f1acb: stopping tablet replica
I20240501 20:00:33.030486 28519 raft_consensus.cc:2229] T 2ddf397fabe8453b8e1601a863a1b84e P 3d7668f070e74286b38fda9d898f1acb [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 20:00:33.030925 28519 raft_consensus.cc:2258] T 2ddf397fabe8453b8e1601a863a1b84e P 3d7668f070e74286b38fda9d898f1acb [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:33.032446 28519 tablet_replica.cc:330] T 14bfbbeb889d43218d5807488de4fed2 P 3d7668f070e74286b38fda9d898f1acb: stopping tablet replica
I20240501 20:00:33.032866 28519 raft_consensus.cc:2229] T 14bfbbeb889d43218d5807488de4fed2 P 3d7668f070e74286b38fda9d898f1acb [term 2 LEADER]: Raft consensus shutting down.
I20240501 20:00:33.033733 28519 raft_consensus.cc:2258] T 14bfbbeb889d43218d5807488de4fed2 P 3d7668f070e74286b38fda9d898f1acb [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:33.035215 28519 tablet_replica.cc:330] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb: stopping tablet replica
I20240501 20:00:33.035609 28519 raft_consensus.cc:2229] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb [term 4 FOLLOWER]: Raft consensus shutting down.
I20240501 20:00:33.036008 28519 raft_consensus.cc:2258] T 21dd29f1917a47f7ab496136a4c7ca07 P 3d7668f070e74286b38fda9d898f1acb [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:33.037539 28519 tablet_replica.cc:330] T 8a0820e7c49f40f89038e0af73567bac P 3d7668f070e74286b38fda9d898f1acb: stopping tablet replica
I20240501 20:00:33.037945 28519 raft_consensus.cc:2229] T 8a0820e7c49f40f89038e0af73567bac P 3d7668f070e74286b38fda9d898f1acb [term 4 FOLLOWER]: Raft consensus shutting down.
I20240501 20:00:33.038421 28519 raft_consensus.cc:2258] T 8a0820e7c49f40f89038e0af73567bac P 3d7668f070e74286b38fda9d898f1acb [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:33.064836 28519 tablet_server.cc:194] TabletServer@127.27.217.195:0 shutdown complete.
I20240501 20:00:33.081650 28519 master.cc:537] Master@127.27.217.254:42645 shutting down...
I20240501 20:00:33.228354 28519 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 6f1eb2bdd9334a72a8e3ff58db174edd [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:00:33.228991 28519 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 6f1eb2bdd9334a72a8e3ff58db174edd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:00:33.229306 28519 tablet_replica.cc:330] T 00000000000000000000000000000000 P 6f1eb2bdd9334a72a8e3ff58db174edd: stopping tablet replica
I20240501 20:00:33.246639 28519 master.cc:552] Master@127.27.217.254:42645 shutdown complete.
3d7668f070e74286b38fda9d898f1acb 1
a068ef6770f14874bd1acb01bce2c329 4
dabdbfb59e8645c297a1696630198b6e 4
3d7668f070e74286b38fda9d898f1acb 3
a068ef6770f14874bd1acb01bce2c329 3
dabdbfb59e8645c297a1696630198b6e 3
3d7668f070e74286b38fda9d898f1acb 1
a068ef6770f14874bd1acb01bce2c329 8
dabdbfb59e8645c297a1696630198b6e 0
3d7668f070e74286b38fda9d898f1acb 3
a068ef6770f14874bd1acb01bce2c329 3
dabdbfb59e8645c297a1696630198b6e 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (30496 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20240501 20:00:33.290627 28519 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.217.254:34681
I20240501 20:00:33.291812 28519 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:00:33.297159 29031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:33.298127 29032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:33.298777 28519 server_base.cc:1034] running on GCE node
W20240501 20:00:33.298751 29034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:33.300001 28519 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:00:33.300244 28519 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:00:33.300390 28519 hybrid_clock.cc:648] HybridClock initialized: now 1714593633300371 us; error 0 us; skew 500 ppm
I20240501 20:00:33.300874 28519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:00:33.303275 28519 webserver.cc:460] Webserver started at http://127.27.217.254:40045/ using document root <none> and password file <none>
I20240501 20:00:33.303747 28519 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:33.303929 28519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:00:33.304206 28519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:00:33.305424 28519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714593602638719-28519-0/minicluster-data/master-0-root/instance:
uuid: "e673238763d6498fbb9641c1f1bcb503"
format_stamp: "Formatted at 2024-05-01 20:00:33 on dist-test-slave-92m0"
I20240501 20:00:33.309902 28519 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 20:00:33.312919 29039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:33.313628 28519 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 20:00:33.313901 28519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714593602638719-28519-0/minicluster-data/master-0-root
uuid: "e673238763d6498fbb9641c1f1bcb503"
format_stamp: "Formatted at 2024-05-01 20:00:33 on dist-test-slave-92m0"
I20240501 20:00:33.314174 28519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714593602638719-28519-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714593602638719-28519-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714593602638719-28519-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:33.354691 28519 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:33.355969 28519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:33.391502 28519 rpc_server.cc:307] RPC server started. Bound to: 127.27.217.254:34681
I20240501 20:00:33.391615 29090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.217.254:34681 every 8 connection(s)
I20240501 20:00:33.395313 29091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:33.404366 29091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e673238763d6498fbb9641c1f1bcb503: Bootstrap starting.
I20240501 20:00:33.408370 29091 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e673238763d6498fbb9641c1f1bcb503: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:33.412281 29091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e673238763d6498fbb9641c1f1bcb503: No bootstrap required, opened a new log
I20240501 20:00:33.414201 29091 raft_consensus.cc:348] T 00000000000000000000000000000000 P e673238763d6498fbb9641c1f1bcb503 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e673238763d6498fbb9641c1f1bcb503" member_type: VOTER }
I20240501 20:00:33.414722 29091 raft_consensus.cc:374] T 00000000000000000000000000000000 P e673238763d6498fbb9641c1f1bcb503 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:33.414969 29091 raft_consensus.cc:729] T 00000000000000000000000000000000 P e673238763d6498fbb9641c1f1bcb503 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e673238763d6498fbb9641c1f1bcb503, State: Initialized, Role: FOLLOWER
I20240501 20:00:33.415490 29091 consensus_queue.cc:260] T 00000000000000000000000000000000 P e673238763d6498fbb9641c1f1bcb503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e673238763d6498fbb9641c1f1bcb503" member_type: VOTER }
I20240501 20:00:33.416002 29091 raft_consensus.cc:388] T 00000000000000000000000000000000 P e673238763d6498fbb9641c1f1bcb503 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:00:33.416306 29091 raft_consensus.cc:482] T 00000000000000000000000000000000 P e673238763d6498fbb9641c1f1bcb503 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:00:33.416581 29091 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e673238763d6498fbb9641c1f1bcb503 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:33.420343 29091 raft_consensus.cc:504] T 00000000000000000000000000000000 P e673238763d6498fbb9641c1f1bcb503 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e673238763d6498fbb9641c1f1bcb503" member_type: VOTER }
I20240501 20:00:33.420816 29091 leader_election.cc:302] T 00000000000000000000000000000000 P e673238763d6498fbb9641c1f1bcb503 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e673238763d6498fbb9641c1f1bcb503; no voters:
I20240501 20:00:33.421952 29091 leader_election.cc:288] T 00000000000000000000000000000000 P e673238763d6498fbb9641c1f1bcb503 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:00:33.422320 29094 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e673238763d6498fbb9641c1f1bcb503 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:00:33.423600 29094 raft_consensus.cc:686] T 00000000000000000000000000000000 P e673238763d6498fbb9641c1f1bcb503 [term 1 LEADER]: Becoming Leader. State: Replica: e673238763d6498fbb9641c1f1bcb503, State: Running, Role: LEADER
I20240501 20:00:33.424307 29094 consensus_queue.cc:237] T 00000000000000000000000000000000 P e673238763d6498fbb9641c1f1bcb503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e673238763d6498fbb9641c1f1bcb503" member_type: VOTER }
I20240501 20:00:33.424857 29091 sys_catalog.cc:564] T 00000000000000000000000000000000 P e673238763d6498fbb9641c1f1bcb503 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:00:33.430689 29096 sys_catalog.cc:455] T 00000000000000000000000000000000 P e673238763d6498fbb9641c1f1bcb503 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e673238763d6498fbb9641c1f1bcb503. Latest consensus state: current_term: 1 leader_uuid: "e673238763d6498fbb9641c1f1bcb503" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e673238763d6498fbb9641c1f1bcb503" member_type: VOTER } }
I20240501 20:00:33.431483 29096 sys_catalog.cc:458] T 00000000000000000000000000000000 P e673238763d6498fbb9641c1f1bcb503 [sys.catalog]: This master's current role is: LEADER
I20240501 20:00:33.432351 29095 sys_catalog.cc:455] T 00000000000000000000000000000000 P e673238763d6498fbb9641c1f1bcb503 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e673238763d6498fbb9641c1f1bcb503" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e673238763d6498fbb9641c1f1bcb503" member_type: VOTER } }
I20240501 20:00:33.433053 29095 sys_catalog.cc:458] T 00000000000000000000000000000000 P e673238763d6498fbb9641c1f1bcb503 [sys.catalog]: This master's current role is: LEADER
I20240501 20:00:33.438880 29101 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:00:33.443760 29101 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:00:33.446942 28519 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 20:00:33.451805 29101 catalog_manager.cc:1347] Generated new cluster ID: f92b1bfc768748228c74707a61b1bed9
I20240501 20:00:33.452147 29101 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:00:33.467216 29101 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:00:33.468485 29101 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:00:33.482098 29101 catalog_manager.cc:5857] T 00000000000000000000000000000000 P e673238763d6498fbb9641c1f1bcb503: Generated new TSK 0
I20240501 20:00:33.482707 29101 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:00:33.513610 28519 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:00:33.519167 29112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:33.520391 29113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:33.522068 28519 server_base.cc:1034] running on GCE node
W20240501 20:00:33.522362 29115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:33.523169 28519 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:00:33.523375 28519 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:00:33.523541 28519 hybrid_clock.cc:648] HybridClock initialized: now 1714593633523523 us; error 0 us; skew 500 ppm
I20240501 20:00:33.524142 28519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:00:33.526468 28519 webserver.cc:460] Webserver started at http://127.27.217.193:34933/ using document root <none> and password file <none>
I20240501 20:00:33.526918 28519 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:33.527083 28519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:00:33.527287 28519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:00:33.528401 28519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714593602638719-28519-0/minicluster-data/ts-0-root/instance:
uuid: "8feebcdef3ba428ba32e62016f3ed365"
format_stamp: "Formatted at 2024-05-01 20:00:33 on dist-test-slave-92m0"
I20240501 20:00:33.532826 28519 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240501 20:00:33.535928 29120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:33.536680 28519 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 20:00:33.536967 28519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714593602638719-28519-0/minicluster-data/ts-0-root
uuid: "8feebcdef3ba428ba32e62016f3ed365"
format_stamp: "Formatted at 2024-05-01 20:00:33 on dist-test-slave-92m0"
I20240501 20:00:33.537230 28519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714593602638719-28519-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714593602638719-28519-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714593602638719-28519-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:33.560940 28519 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:33.562142 28519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:33.563565 28519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:00:33.565719 28519 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:00:33.565924 28519 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:33.566143 28519 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:00:33.566306 28519 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:33.604003 28519 rpc_server.cc:307] RPC server started. Bound to: 127.27.217.193:43937
I20240501 20:00:33.604135 29182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.217.193:43937 every 8 connection(s)
I20240501 20:00:33.608371 28519 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:00:33.616147 29187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:33.618495 29188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:33.621553 29190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:33.622211 29183 heartbeater.cc:347] Connected to a master server at 127.27.217.254:34681
I20240501 20:00:33.622637 29183 heartbeater.cc:464] Registering TS with master...
I20240501 20:00:33.623349 28519 server_base.cc:1034] running on GCE node
I20240501 20:00:33.623492 29183 heartbeater.cc:511] Master 127.27.217.254:34681 requested a full tablet report, sending...
I20240501 20:00:33.624670 28519 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:00:33.624884 28519 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:00:33.625089 28519 hybrid_clock.cc:648] HybridClock initialized: now 1714593633625071 us; error 0 us; skew 500 ppm
I20240501 20:00:33.625609 29056 ts_manager.cc:194] Registered new tserver with Master: 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193:43937)
I20240501 20:00:33.625916 28519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:00:33.628417 29056 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50744
I20240501 20:00:33.628788 28519 webserver.cc:460] Webserver started at http://127.27.217.194:42393/ using document root <none> and password file <none>
I20240501 20:00:33.629453 28519 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:33.629658 28519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:00:33.629933 28519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:00:33.631036 28519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714593602638719-28519-0/minicluster-data/ts-1-root/instance:
uuid: "78bf5e3ff675477cb9f9c874292fa63f"
format_stamp: "Formatted at 2024-05-01 20:00:33 on dist-test-slave-92m0"
I20240501 20:00:33.635217 28519 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240501 20:00:33.638439 29195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:33.639263 28519 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240501 20:00:33.639536 28519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714593602638719-28519-0/minicluster-data/ts-1-root
uuid: "78bf5e3ff675477cb9f9c874292fa63f"
format_stamp: "Formatted at 2024-05-01 20:00:33 on dist-test-slave-92m0"
I20240501 20:00:33.639803 28519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714593602638719-28519-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714593602638719-28519-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714593602638719-28519-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:33.659821 28519 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:33.661712 28519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:33.663131 28519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:00:33.665367 28519 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:00:33.665576 28519 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:33.665819 28519 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:00:33.665979 28519 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:33.702452 28519 rpc_server.cc:307] RPC server started. Bound to: 127.27.217.194:39263
I20240501 20:00:33.702560 29257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.217.194:39263 every 8 connection(s)
I20240501 20:00:33.706832 28519 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:00:33.713651 29261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:33.716173 29262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:00:33.717607 29264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:00:33.717803 28519 server_base.cc:1034] running on GCE node
I20240501 20:00:33.718911 28519 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240501 20:00:33.719038 29258 heartbeater.cc:347] Connected to a master server at 127.27.217.254:34681
W20240501 20:00:33.719178 28519 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:00:33.719372 28519 hybrid_clock.cc:648] HybridClock initialized: now 1714593633719359 us; error 0 us; skew 500 ppm
I20240501 20:00:33.719426 29258 heartbeater.cc:464] Registering TS with master...
I20240501 20:00:33.719967 28519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:00:33.720211 29258 heartbeater.cc:511] Master 127.27.217.254:34681 requested a full tablet report, sending...
I20240501 20:00:33.722255 29056 ts_manager.cc:194] Registered new tserver with Master: 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194:39263)
I20240501 20:00:33.722772 28519 webserver.cc:460] Webserver started at http://127.27.217.195:33473/ using document root <none> and password file <none>
I20240501 20:00:33.723260 28519 fs_manager.cc:359] Metadata directory not provided
I20240501 20:00:33.723456 28519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:00:33.723692 28519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:00:33.723693 29056 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50760
I20240501 20:00:33.725211 28519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714593602638719-28519-0/minicluster-data/ts-2-root/instance:
uuid: "16f5b5441d8c4a44a9c0a6cca8356822"
format_stamp: "Formatted at 2024-05-01 20:00:33 on dist-test-slave-92m0"
I20240501 20:00:33.729729 28519 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 20:00:33.732997 29269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:33.733846 28519 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 20:00:33.734141 28519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714593602638719-28519-0/minicluster-data/ts-2-root
uuid: "16f5b5441d8c4a44a9c0a6cca8356822"
format_stamp: "Formatted at 2024-05-01 20:00:33 on dist-test-slave-92m0"
I20240501 20:00:33.734431 28519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714593602638719-28519-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714593602638719-28519-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1714593602638719-28519-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:00:33.763099 28519 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:00:33.764273 28519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:00:33.765651 28519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:00:33.767685 28519 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:00:33.767880 28519 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:33.768151 28519 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:00:33.768312 28519 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:00:33.805810 28519 rpc_server.cc:307] RPC server started. Bound to: 127.27.217.195:37025
I20240501 20:00:33.805895 29331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.217.195:37025 every 8 connection(s)
I20240501 20:00:33.818239 29332 heartbeater.cc:347] Connected to a master server at 127.27.217.254:34681
I20240501 20:00:33.818609 29332 heartbeater.cc:464] Registering TS with master...
I20240501 20:00:33.819314 29332 heartbeater.cc:511] Master 127.27.217.254:34681 requested a full tablet report, sending...
I20240501 20:00:33.820991 29056 ts_manager.cc:194] Registered new tserver with Master: 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195:37025)
I20240501 20:00:33.821923 28519 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013108643s
I20240501 20:00:33.822273 29056 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50774
I20240501 20:00:33.824188 28519 test_util.cc:268] Using random seed: -260411991
I20240501 20:00:33.845213 29056 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50784:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 20:00:33.847950 29056 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 20:00:33.919287 29297 tablet_service.cc:1467] Processing CreateTablet for tablet 0d9b61e9c98e4d19b9272fe70fab0c3d (DEFAULT_TABLE table=test-workload [id=43ed327d552c4f429b090f1a8f3810f3]), partition=RANGE (key) PARTITION VALUES < 214748364
I20240501 20:00:33.920718 29297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d9b61e9c98e4d19b9272fe70fab0c3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:33.924394 29296 tablet_service.cc:1467] Processing CreateTablet for tablet 3d51e8d55df849e5ac6e5acdc42ac63c (DEFAULT_TABLE table=test-workload [id=43ed327d552c4f429b090f1a8f3810f3]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20240501 20:00:33.925769 29296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d51e8d55df849e5ac6e5acdc42ac63c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:33.930051 29295 tablet_service.cc:1467] Processing CreateTablet for tablet f4895041b0924276aa587adbb8dc9e98 (DEFAULT_TABLE table=test-workload [id=43ed327d552c4f429b090f1a8f3810f3]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20240501 20:00:33.931382 29295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4895041b0924276aa587adbb8dc9e98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:33.935997 29294 tablet_service.cc:1467] Processing CreateTablet for tablet afbcfea6a7894b5ab7c122d3d4134869 (DEFAULT_TABLE table=test-workload [id=43ed327d552c4f429b090f1a8f3810f3]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20240501 20:00:33.937332 29294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afbcfea6a7894b5ab7c122d3d4134869. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:33.938829 29223 tablet_service.cc:1467] Processing CreateTablet for tablet 0d9b61e9c98e4d19b9272fe70fab0c3d (DEFAULT_TABLE table=test-workload [id=43ed327d552c4f429b090f1a8f3810f3]), partition=RANGE (key) PARTITION VALUES < 214748364
I20240501 20:00:33.940117 29223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d9b61e9c98e4d19b9272fe70fab0c3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:33.941282 29292 tablet_service.cc:1467] Processing CreateTablet for tablet 27c72ce2560f4d3baf61f0441f5c4129 (DEFAULT_TABLE table=test-workload [id=43ed327d552c4f429b090f1a8f3810f3]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20240501 20:00:33.943229 29222 tablet_service.cc:1467] Processing CreateTablet for tablet 3d51e8d55df849e5ac6e5acdc42ac63c (DEFAULT_TABLE table=test-workload [id=43ed327d552c4f429b090f1a8f3810f3]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20240501 20:00:33.943513 29291 tablet_service.cc:1467] Processing CreateTablet for tablet 419e173751fc4cd18b50341160a84d24 (DEFAULT_TABLE table=test-workload [id=43ed327d552c4f429b090f1a8f3810f3]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20240501 20:00:33.944527 29222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d51e8d55df849e5ac6e5acdc42ac63c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:33.945755 29217 tablet_service.cc:1467] Processing CreateTablet for tablet 419e173751fc4cd18b50341160a84d24 (DEFAULT_TABLE table=test-workload [id=43ed327d552c4f429b090f1a8f3810f3]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20240501 20:00:33.947114 29217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 419e173751fc4cd18b50341160a84d24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:33.946990 29289 tablet_service.cc:1467] Processing CreateTablet for tablet eed1ac06c82147e3bb681f4bcd1d8688 (DEFAULT_TABLE table=test-workload [id=43ed327d552c4f429b090f1a8f3810f3]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20240501 20:00:33.939833 29293 tablet_service.cc:1467] Processing CreateTablet for tablet 5bd59b3e54144bd2986538a355cadd9b (DEFAULT_TABLE table=test-workload [id=43ed327d552c4f429b090f1a8f3810f3]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20240501 20:00:33.949247 29288 tablet_service.cc:1467] Processing CreateTablet for tablet 6c7b0c234f7d4cc0a0848e47e24c5d8b (DEFAULT_TABLE table=test-workload [id=43ed327d552c4f429b090f1a8f3810f3]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20240501 20:00:33.945716 29290 tablet_service.cc:1467] Processing CreateTablet for tablet d057ab88d0084efe8d2fe408a42a24b7 (DEFAULT_TABLE table=test-workload [id=43ed327d552c4f429b090f1a8f3810f3]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20240501 20:00:33.949283 29221 tablet_service.cc:1467] Processing CreateTablet for tablet f4895041b0924276aa587adbb8dc9e98 (DEFAULT_TABLE table=test-workload [id=43ed327d552c4f429b090f1a8f3810f3]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20240501 20:00:33.951769 29221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4895041b0924276aa587adbb8dc9e98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:33.951583 29219 tablet_service.cc:1467] Processing CreateTablet for tablet 5bd59b3e54144bd2986538a355cadd9b (DEFAULT_TABLE table=test-workload [id=43ed327d552c4f429b090f1a8f3810f3]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20240501 20:00:33.954587 29292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27c72ce2560f4d3baf61f0441f5c4129. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:33.954051 29219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bd59b3e54144bd2986538a355cadd9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:33.957334 29293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bd59b3e54144bd2986538a355cadd9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:33.950423 29220 tablet_service.cc:1467] Processing CreateTablet for tablet afbcfea6a7894b5ab7c122d3d4134869 (DEFAULT_TABLE table=test-workload [id=43ed327d552c4f429b090f1a8f3810f3]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20240501 20:00:33.960837 29216 tablet_service.cc:1467] Processing CreateTablet for tablet d057ab88d0084efe8d2fe408a42a24b7 (DEFAULT_TABLE table=test-workload [id=43ed327d552c4f429b090f1a8f3810f3]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20240501 20:00:33.961102 29148 tablet_service.cc:1467] Processing CreateTablet for tablet 0d9b61e9c98e4d19b9272fe70fab0c3d (DEFAULT_TABLE table=test-workload [id=43ed327d552c4f429b090f1a8f3810f3]), partition=RANGE (key) PARTITION VALUES < 214748364
I20240501 20:00:33.961243 29218 tablet_service.cc:1467] Processing CreateTablet for tablet 27c72ce2560f4d3baf61f0441f5c4129 (DEFAULT_TABLE table=test-workload [id=43ed327d552c4f429b090f1a8f3810f3]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20240501 20:00:33.962472 29148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d9b61e9c98e4d19b9272fe70fab0c3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:33.964164 29147 tablet_service.cc:1467] Processing CreateTablet for tablet 3d51e8d55df849e5ac6e5acdc42ac63c (DEFAULT_TABLE table=test-workload [id=43ed327d552c4f429b090f1a8f3810f3]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20240501 20:00:33.965476 29147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d51e8d55df849e5ac6e5acdc42ac63c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:33.966312 29140 tablet_service.cc:1467] Processing CreateTablet for tablet eed1ac06c82147e3bb681f4bcd1d8688 (DEFAULT_TABLE table=test-workload [id=43ed327d552c4f429b090f1a8f3810f3]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20240501 20:00:33.967528 29140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eed1ac06c82147e3bb681f4bcd1d8688. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:33.969328 29145 tablet_service.cc:1467] Processing CreateTablet for tablet afbcfea6a7894b5ab7c122d3d4134869 (DEFAULT_TABLE table=test-workload [id=43ed327d552c4f429b090f1a8f3810f3]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20240501 20:00:33.969640 29139 tablet_service.cc:1467] Processing CreateTablet for tablet 6c7b0c234f7d4cc0a0848e47e24c5d8b (DEFAULT_TABLE table=test-workload [id=43ed327d552c4f429b090f1a8f3810f3]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20240501 20:00:33.970618 29145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afbcfea6a7894b5ab7c122d3d4134869. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:33.971029 29216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d057ab88d0084efe8d2fe408a42a24b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:33.971655 29139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c7b0c234f7d4cc0a0848e47e24c5d8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:33.972688 29144 tablet_service.cc:1467] Processing CreateTablet for tablet 5bd59b3e54144bd2986538a355cadd9b (DEFAULT_TABLE table=test-workload [id=43ed327d552c4f429b090f1a8f3810f3]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20240501 20:00:33.973985 29144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bd59b3e54144bd2986538a355cadd9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:33.976763 29288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c7b0c234f7d4cc0a0848e47e24c5d8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:33.977476 29143 tablet_service.cc:1467] Processing CreateTablet for tablet 27c72ce2560f4d3baf61f0441f5c4129 (DEFAULT_TABLE table=test-workload [id=43ed327d552c4f429b090f1a8f3810f3]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20240501 20:00:33.978749 29143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27c72ce2560f4d3baf61f0441f5c4129. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:33.980330 29290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d057ab88d0084efe8d2fe408a42a24b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:33.982090 29146 tablet_service.cc:1467] Processing CreateTablet for tablet f4895041b0924276aa587adbb8dc9e98 (DEFAULT_TABLE table=test-workload [id=43ed327d552c4f429b090f1a8f3810f3]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20240501 20:00:33.983420 29146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4895041b0924276aa587adbb8dc9e98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:33.984354 29291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 419e173751fc4cd18b50341160a84d24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:33.985172 29142 tablet_service.cc:1467] Processing CreateTablet for tablet 419e173751fc4cd18b50341160a84d24 (DEFAULT_TABLE table=test-workload [id=43ed327d552c4f429b090f1a8f3810f3]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20240501 20:00:33.987383 29141 tablet_service.cc:1467] Processing CreateTablet for tablet d057ab88d0084efe8d2fe408a42a24b7 (DEFAULT_TABLE table=test-workload [id=43ed327d552c4f429b090f1a8f3810f3]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20240501 20:00:33.988729 29142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 419e173751fc4cd18b50341160a84d24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:33.993639 29289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eed1ac06c82147e3bb681f4bcd1d8688. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:33.994637 29141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d057ab88d0084efe8d2fe408a42a24b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:33.996879 29218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27c72ce2560f4d3baf61f0441f5c4129. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:34.004557 29220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afbcfea6a7894b5ab7c122d3d4134869. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:34.010818 29215 tablet_service.cc:1467] Processing CreateTablet for tablet eed1ac06c82147e3bb681f4bcd1d8688 (DEFAULT_TABLE table=test-workload [id=43ed327d552c4f429b090f1a8f3810f3]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20240501 20:00:34.012267 29215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eed1ac06c82147e3bb681f4bcd1d8688. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:34.023805 29214 tablet_service.cc:1467] Processing CreateTablet for tablet 6c7b0c234f7d4cc0a0848e47e24c5d8b (DEFAULT_TABLE table=test-workload [id=43ed327d552c4f429b090f1a8f3810f3]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20240501 20:00:34.036742 29214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c7b0c234f7d4cc0a0848e47e24c5d8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:00:34.056224 29347 tablet_bootstrap.cc:492] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 16f5b5441d8c4a44a9c0a6cca8356822: Bootstrap starting.
I20240501 20:00:34.062513 29347 tablet_bootstrap.cc:654] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 16f5b5441d8c4a44a9c0a6cca8356822: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:34.091883 29348 tablet_bootstrap.cc:492] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 78bf5e3ff675477cb9f9c874292fa63f: Bootstrap starting.
I20240501 20:00:34.107928 29348 tablet_bootstrap.cc:654] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 78bf5e3ff675477cb9f9c874292fa63f: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:34.127646 29349 tablet_bootstrap.cc:492] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365: Bootstrap starting.
I20240501 20:00:34.128962 29348 tablet_bootstrap.cc:492] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 78bf5e3ff675477cb9f9c874292fa63f: No bootstrap required, opened a new log
I20240501 20:00:34.129539 29348 ts_tablet_manager.cc:1367] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 78bf5e3ff675477cb9f9c874292fa63f: Time spent bootstrapping tablet: real 0.038s user 0.014s sys 0.006s
I20240501 20:00:34.134474 29348 raft_consensus.cc:348] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } } peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } }
I20240501 20:00:34.136608 29348 raft_consensus.cc:374] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:34.136866 29348 raft_consensus.cc:729] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78bf5e3ff675477cb9f9c874292fa63f, State: Initialized, Role: FOLLOWER
I20240501 20:00:34.137445 29348 consensus_queue.cc:260] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 78bf5e3ff675477cb9f9c874292fa63f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } } peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } }
I20240501 20:00:34.139645 29258 heartbeater.cc:503] Master 127.27.217.254:34681 was elected leader, sending a full tablet report...
I20240501 20:00:34.142412 29347 tablet_bootstrap.cc:492] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 16f5b5441d8c4a44a9c0a6cca8356822: No bootstrap required, opened a new log
I20240501 20:00:34.142541 29349 tablet_bootstrap.cc:654] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:34.142946 29347 ts_tablet_manager.cc:1367] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 16f5b5441d8c4a44a9c0a6cca8356822: Time spent bootstrapping tablet: real 0.088s user 0.012s sys 0.003s
I20240501 20:00:34.146781 29348 ts_tablet_manager.cc:1398] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 78bf5e3ff675477cb9f9c874292fa63f: Time spent starting tablet: real 0.016s user 0.004s sys 0.001s
I20240501 20:00:34.147500 29349 tablet_bootstrap.cc:492] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365: No bootstrap required, opened a new log
I20240501 20:00:34.144991 29347 raft_consensus.cc:348] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } } peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } }
I20240501 20:00:34.148125 29347 raft_consensus.cc:374] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:34.148252 29349 ts_tablet_manager.cc:1367] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.007s
I20240501 20:00:34.148465 29347 raft_consensus.cc:729] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16f5b5441d8c4a44a9c0a6cca8356822, State: Initialized, Role: FOLLOWER
I20240501 20:00:34.148610 29348 tablet_bootstrap.cc:492] T 3d51e8d55df849e5ac6e5acdc42ac63c P 78bf5e3ff675477cb9f9c874292fa63f: Bootstrap starting.
I20240501 20:00:34.149318 29347 consensus_queue.cc:260] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 16f5b5441d8c4a44a9c0a6cca8356822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } } peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } }
I20240501 20:00:34.150997 29349 raft_consensus.cc:348] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.151798 29349 raft_consensus.cc:374] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:34.152103 29349 raft_consensus.cc:729] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8feebcdef3ba428ba32e62016f3ed365, State: Initialized, Role: FOLLOWER
I20240501 20:00:34.152876 29349 consensus_queue.cc:260] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.156234 29348 tablet_bootstrap.cc:654] T 3d51e8d55df849e5ac6e5acdc42ac63c P 78bf5e3ff675477cb9f9c874292fa63f: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:34.158409 29347 ts_tablet_manager.cc:1398] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 16f5b5441d8c4a44a9c0a6cca8356822: Time spent starting tablet: real 0.015s user 0.006s sys 0.007s
I20240501 20:00:34.159987 29349 ts_tablet_manager.cc:1398] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365: Time spent starting tablet: real 0.011s user 0.005s sys 0.004s
I20240501 20:00:34.160787 29332 heartbeater.cc:503] Master 127.27.217.254:34681 was elected leader, sending a full tablet report...
I20240501 20:00:34.160877 29183 heartbeater.cc:503] Master 127.27.217.254:34681 was elected leader, sending a full tablet report...
I20240501 20:00:34.161672 29348 tablet_bootstrap.cc:492] T 3d51e8d55df849e5ac6e5acdc42ac63c P 78bf5e3ff675477cb9f9c874292fa63f: No bootstrap required, opened a new log
I20240501 20:00:34.162169 29348 ts_tablet_manager.cc:1367] T 3d51e8d55df849e5ac6e5acdc42ac63c P 78bf5e3ff675477cb9f9c874292fa63f: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20240501 20:00:34.165171 29347 tablet_bootstrap.cc:492] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822: Bootstrap starting.
I20240501 20:00:34.168432 29348 raft_consensus.cc:348] T 3d51e8d55df849e5ac6e5acdc42ac63c P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.169004 29349 tablet_bootstrap.cc:492] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 8feebcdef3ba428ba32e62016f3ed365: Bootstrap starting.
I20240501 20:00:34.169265 29348 raft_consensus.cc:374] T 3d51e8d55df849e5ac6e5acdc42ac63c P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:34.169600 29348 raft_consensus.cc:729] T 3d51e8d55df849e5ac6e5acdc42ac63c P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78bf5e3ff675477cb9f9c874292fa63f, State: Initialized, Role: FOLLOWER
I20240501 20:00:34.170370 29348 consensus_queue.cc:260] T 3d51e8d55df849e5ac6e5acdc42ac63c P 78bf5e3ff675477cb9f9c874292fa63f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.171509 29347 tablet_bootstrap.cc:654] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:34.174336 29348 ts_tablet_manager.cc:1398] T 3d51e8d55df849e5ac6e5acdc42ac63c P 78bf5e3ff675477cb9f9c874292fa63f: Time spent starting tablet: real 0.012s user 0.004s sys 0.004s
I20240501 20:00:34.175071 29349 tablet_bootstrap.cc:654] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 8feebcdef3ba428ba32e62016f3ed365: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:34.175544 29348 tablet_bootstrap.cc:492] T d057ab88d0084efe8d2fe408a42a24b7 P 78bf5e3ff675477cb9f9c874292fa63f: Bootstrap starting.
I20240501 20:00:34.184036 29348 tablet_bootstrap.cc:654] T d057ab88d0084efe8d2fe408a42a24b7 P 78bf5e3ff675477cb9f9c874292fa63f: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:34.202666 29348 tablet_bootstrap.cc:492] T d057ab88d0084efe8d2fe408a42a24b7 P 78bf5e3ff675477cb9f9c874292fa63f: No bootstrap required, opened a new log
I20240501 20:00:34.205032 29348 ts_tablet_manager.cc:1367] T d057ab88d0084efe8d2fe408a42a24b7 P 78bf5e3ff675477cb9f9c874292fa63f: Time spent bootstrapping tablet: real 0.030s user 0.017s sys 0.004s
I20240501 20:00:34.208587 29348 raft_consensus.cc:348] T d057ab88d0084efe8d2fe408a42a24b7 P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.209604 29348 raft_consensus.cc:374] T d057ab88d0084efe8d2fe408a42a24b7 P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:34.209983 29348 raft_consensus.cc:729] T d057ab88d0084efe8d2fe408a42a24b7 P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78bf5e3ff675477cb9f9c874292fa63f, State: Initialized, Role: FOLLOWER
I20240501 20:00:34.210856 29348 consensus_queue.cc:260] T d057ab88d0084efe8d2fe408a42a24b7 P 78bf5e3ff675477cb9f9c874292fa63f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.214223 29348 ts_tablet_manager.cc:1398] T d057ab88d0084efe8d2fe408a42a24b7 P 78bf5e3ff675477cb9f9c874292fa63f: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20240501 20:00:34.220362 29348 tablet_bootstrap.cc:492] T afbcfea6a7894b5ab7c122d3d4134869 P 78bf5e3ff675477cb9f9c874292fa63f: Bootstrap starting.
I20240501 20:00:34.228292 29348 tablet_bootstrap.cc:654] T afbcfea6a7894b5ab7c122d3d4134869 P 78bf5e3ff675477cb9f9c874292fa63f: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:34.240625 29348 tablet_bootstrap.cc:492] T afbcfea6a7894b5ab7c122d3d4134869 P 78bf5e3ff675477cb9f9c874292fa63f: No bootstrap required, opened a new log
I20240501 20:00:34.241115 29348 ts_tablet_manager.cc:1367] T afbcfea6a7894b5ab7c122d3d4134869 P 78bf5e3ff675477cb9f9c874292fa63f: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.008s
I20240501 20:00:34.243710 29348 raft_consensus.cc:348] T afbcfea6a7894b5ab7c122d3d4134869 P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.244493 29348 raft_consensus.cc:374] T afbcfea6a7894b5ab7c122d3d4134869 P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:34.244805 29348 raft_consensus.cc:729] T afbcfea6a7894b5ab7c122d3d4134869 P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78bf5e3ff675477cb9f9c874292fa63f, State: Initialized, Role: FOLLOWER
I20240501 20:00:34.245522 29348 consensus_queue.cc:260] T afbcfea6a7894b5ab7c122d3d4134869 P 78bf5e3ff675477cb9f9c874292fa63f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.246510 29349 tablet_bootstrap.cc:492] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 8feebcdef3ba428ba32e62016f3ed365: No bootstrap required, opened a new log
I20240501 20:00:34.247025 29349 ts_tablet_manager.cc:1367] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 8feebcdef3ba428ba32e62016f3ed365: Time spent bootstrapping tablet: real 0.078s user 0.030s sys 0.037s
I20240501 20:00:34.247220 29347 tablet_bootstrap.cc:492] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822: No bootstrap required, opened a new log
I20240501 20:00:34.247710 29347 ts_tablet_manager.cc:1367] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822: Time spent bootstrapping tablet: real 0.083s user 0.036s sys 0.042s
I20240501 20:00:34.247745 29348 ts_tablet_manager.cc:1398] T afbcfea6a7894b5ab7c122d3d4134869 P 78bf5e3ff675477cb9f9c874292fa63f: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240501 20:00:34.248939 29348 tablet_bootstrap.cc:492] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f: Bootstrap starting.
I20240501 20:00:34.250053 29349 raft_consensus.cc:348] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.250854 29349 raft_consensus.cc:374] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:34.250490 29347 raft_consensus.cc:348] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.251246 29349 raft_consensus.cc:729] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8feebcdef3ba428ba32e62016f3ed365, State: Initialized, Role: FOLLOWER
I20240501 20:00:34.251371 29347 raft_consensus.cc:374] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:34.251912 29347 raft_consensus.cc:729] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16f5b5441d8c4a44a9c0a6cca8356822, State: Initialized, Role: FOLLOWER
I20240501 20:00:34.252424 29349 consensus_queue.cc:260] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 8feebcdef3ba428ba32e62016f3ed365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.252774 29347 consensus_queue.cc:260] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.254995 29348 tablet_bootstrap.cc:654] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:34.255033 29349 ts_tablet_manager.cc:1398] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 8feebcdef3ba428ba32e62016f3ed365: Time spent starting tablet: real 0.008s user 0.003s sys 0.002s
I20240501 20:00:34.256289 29349 tablet_bootstrap.cc:492] T 5bd59b3e54144bd2986538a355cadd9b P 8feebcdef3ba428ba32e62016f3ed365: Bootstrap starting.
I20240501 20:00:34.256690 29347 ts_tablet_manager.cc:1398] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822: Time spent starting tablet: real 0.008s user 0.002s sys 0.004s
I20240501 20:00:34.258780 29347 tablet_bootstrap.cc:492] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822: Bootstrap starting.
I20240501 20:00:34.264024 29349 tablet_bootstrap.cc:654] T 5bd59b3e54144bd2986538a355cadd9b P 8feebcdef3ba428ba32e62016f3ed365: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:34.264425 29347 tablet_bootstrap.cc:654] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:34.267105 29354 raft_consensus.cc:482] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:34.267604 29354 raft_consensus.cc:504] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.270084 29354 leader_election.cc:288] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193:43937), 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194:39263)
I20240501 20:00:34.280177 29347 tablet_bootstrap.cc:492] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822: No bootstrap required, opened a new log
I20240501 20:00:34.280683 29347 ts_tablet_manager.cc:1367] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.010s
I20240501 20:00:34.283223 29347 raft_consensus.cc:348] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.283967 29347 raft_consensus.cc:374] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:34.284283 29347 raft_consensus.cc:729] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16f5b5441d8c4a44a9c0a6cca8356822, State: Initialized, Role: FOLLOWER
I20240501 20:00:34.285135 29347 consensus_queue.cc:260] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.287485 29347 ts_tablet_manager.cc:1398] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240501 20:00:34.288436 29347 tablet_bootstrap.cc:492] T afbcfea6a7894b5ab7c122d3d4134869 P 16f5b5441d8c4a44a9c0a6cca8356822: Bootstrap starting.
I20240501 20:00:34.294154 29347 tablet_bootstrap.cc:654] T afbcfea6a7894b5ab7c122d3d4134869 P 16f5b5441d8c4a44a9c0a6cca8356822: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:34.301774 29348 tablet_bootstrap.cc:492] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f: No bootstrap required, opened a new log
I20240501 20:00:34.302244 29348 ts_tablet_manager.cc:1367] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f: Time spent bootstrapping tablet: real 0.053s user 0.018s sys 0.032s
I20240501 20:00:34.302296 29347 tablet_bootstrap.cc:492] T afbcfea6a7894b5ab7c122d3d4134869 P 16f5b5441d8c4a44a9c0a6cca8356822: No bootstrap required, opened a new log
I20240501 20:00:34.302899 29347 ts_tablet_manager.cc:1367] T afbcfea6a7894b5ab7c122d3d4134869 P 16f5b5441d8c4a44a9c0a6cca8356822: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.004s
I20240501 20:00:34.305509 29347 raft_consensus.cc:348] T afbcfea6a7894b5ab7c122d3d4134869 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.305500 29348 raft_consensus.cc:348] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.306272 29347 raft_consensus.cc:374] T afbcfea6a7894b5ab7c122d3d4134869 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:34.306488 29348 raft_consensus.cc:374] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:34.306596 29347 raft_consensus.cc:729] T afbcfea6a7894b5ab7c122d3d4134869 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16f5b5441d8c4a44a9c0a6cca8356822, State: Initialized, Role: FOLLOWER
I20240501 20:00:34.306881 29348 raft_consensus.cc:729] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78bf5e3ff675477cb9f9c874292fa63f, State: Initialized, Role: FOLLOWER
I20240501 20:00:34.307171 29347 consensus_queue.cc:260] T afbcfea6a7894b5ab7c122d3d4134869 P 16f5b5441d8c4a44a9c0a6cca8356822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.307744 29348 consensus_queue.cc:260] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.310150 29348 ts_tablet_manager.cc:1398] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240501 20:00:34.310518 29233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d51e8d55df849e5ac6e5acdc42ac63c" candidate_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78bf5e3ff675477cb9f9c874292fa63f" is_pre_election: true
I20240501 20:00:34.311110 29348 tablet_bootstrap.cc:492] T f4895041b0924276aa587adbb8dc9e98 P 78bf5e3ff675477cb9f9c874292fa63f: Bootstrap starting.
I20240501 20:00:34.311278 29347 ts_tablet_manager.cc:1398] T afbcfea6a7894b5ab7c122d3d4134869 P 16f5b5441d8c4a44a9c0a6cca8356822: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240501 20:00:34.311292 29233 raft_consensus.cc:2454] T 3d51e8d55df849e5ac6e5acdc42ac63c P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16f5b5441d8c4a44a9c0a6cca8356822 in term 0.
I20240501 20:00:34.312399 29347 tablet_bootstrap.cc:492] T 27c72ce2560f4d3baf61f0441f5c4129 P 16f5b5441d8c4a44a9c0a6cca8356822: Bootstrap starting.
I20240501 20:00:34.313582 29158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d51e8d55df849e5ac6e5acdc42ac63c" candidate_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8feebcdef3ba428ba32e62016f3ed365" is_pre_election: true
I20240501 20:00:34.318511 29347 tablet_bootstrap.cc:654] T 27c72ce2560f4d3baf61f0441f5c4129 P 16f5b5441d8c4a44a9c0a6cca8356822: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:34.318842 29348 tablet_bootstrap.cc:654] T f4895041b0924276aa587adbb8dc9e98 P 78bf5e3ff675477cb9f9c874292fa63f: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:34.325585 29273 leader_election.cc:302] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16f5b5441d8c4a44a9c0a6cca8356822, 78bf5e3ff675477cb9f9c874292fa63f; no voters:
I20240501 20:00:34.326802 29354 raft_consensus.cc:2781] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240501 20:00:34.326984 29273 leader_election.cc:341] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193:43937): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:00:34.327188 29354 raft_consensus.cc:482] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:00:34.327654 29354 raft_consensus.cc:3037] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:34.331981 29349 tablet_bootstrap.cc:492] T 5bd59b3e54144bd2986538a355cadd9b P 8feebcdef3ba428ba32e62016f3ed365: No bootstrap required, opened a new log
I20240501 20:00:34.332528 29349 ts_tablet_manager.cc:1367] T 5bd59b3e54144bd2986538a355cadd9b P 8feebcdef3ba428ba32e62016f3ed365: Time spent bootstrapping tablet: real 0.076s user 0.014s sys 0.003s
I20240501 20:00:34.334517 29348 tablet_bootstrap.cc:492] T f4895041b0924276aa587adbb8dc9e98 P 78bf5e3ff675477cb9f9c874292fa63f: No bootstrap required, opened a new log
I20240501 20:00:34.335031 29348 ts_tablet_manager.cc:1367] T f4895041b0924276aa587adbb8dc9e98 P 78bf5e3ff675477cb9f9c874292fa63f: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.015s
I20240501 20:00:34.335175 29349 raft_consensus.cc:348] T 5bd59b3e54144bd2986538a355cadd9b P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.336146 29349 raft_consensus.cc:374] T 5bd59b3e54144bd2986538a355cadd9b P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:34.336447 29349 raft_consensus.cc:729] T 5bd59b3e54144bd2986538a355cadd9b P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8feebcdef3ba428ba32e62016f3ed365, State: Initialized, Role: FOLLOWER
I20240501 20:00:34.337221 29349 consensus_queue.cc:260] T 5bd59b3e54144bd2986538a355cadd9b P 8feebcdef3ba428ba32e62016f3ed365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.338352 29348 raft_consensus.cc:348] T f4895041b0924276aa587adbb8dc9e98 P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.339385 29348 raft_consensus.cc:374] T f4895041b0924276aa587adbb8dc9e98 P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:34.339705 29348 raft_consensus.cc:729] T f4895041b0924276aa587adbb8dc9e98 P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78bf5e3ff675477cb9f9c874292fa63f, State: Initialized, Role: FOLLOWER
I20240501 20:00:34.340139 29349 ts_tablet_manager.cc:1398] T 5bd59b3e54144bd2986538a355cadd9b P 8feebcdef3ba428ba32e62016f3ed365: Time spent starting tablet: real 0.007s user 0.002s sys 0.005s
I20240501 20:00:34.340577 29348 consensus_queue.cc:260] T f4895041b0924276aa587adbb8dc9e98 P 78bf5e3ff675477cb9f9c874292fa63f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.341208 29349 tablet_bootstrap.cc:492] T eed1ac06c82147e3bb681f4bcd1d8688 P 8feebcdef3ba428ba32e62016f3ed365: Bootstrap starting.
I20240501 20:00:34.342159 29354 raft_consensus.cc:504] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.347069 29349 tablet_bootstrap.cc:654] T eed1ac06c82147e3bb681f4bcd1d8688 P 8feebcdef3ba428ba32e62016f3ed365: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:34.350396 29158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d51e8d55df849e5ac6e5acdc42ac63c" candidate_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8feebcdef3ba428ba32e62016f3ed365"
W20240501 20:00:34.352119 29273 leader_election.cc:341] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193:43937): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:00:34.353914 29369 raft_consensus.cc:482] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:34.354357 29369 raft_consensus.cc:504] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.350418 29348 ts_tablet_manager.cc:1398] T f4895041b0924276aa587adbb8dc9e98 P 78bf5e3ff675477cb9f9c874292fa63f: Time spent starting tablet: real 0.015s user 0.005s sys 0.001s
I20240501 20:00:34.355691 29233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d51e8d55df849e5ac6e5acdc42ac63c" candidate_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78bf5e3ff675477cb9f9c874292fa63f"
I20240501 20:00:34.356204 29369 leader_election.cc:288] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193:43937), 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194:39263)
I20240501 20:00:34.356282 29233 raft_consensus.cc:3037] T 3d51e8d55df849e5ac6e5acdc42ac63c P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:34.357853 29348 tablet_bootstrap.cc:492] T 419e173751fc4cd18b50341160a84d24 P 78bf5e3ff675477cb9f9c874292fa63f: Bootstrap starting.
I20240501 20:00:34.358098 29349 tablet_bootstrap.cc:492] T eed1ac06c82147e3bb681f4bcd1d8688 P 8feebcdef3ba428ba32e62016f3ed365: No bootstrap required, opened a new log
I20240501 20:00:34.358589 29349 ts_tablet_manager.cc:1367] T eed1ac06c82147e3bb681f4bcd1d8688 P 8feebcdef3ba428ba32e62016f3ed365: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.005s
I20240501 20:00:34.361263 29349 raft_consensus.cc:348] T eed1ac06c82147e3bb681f4bcd1d8688 P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.361995 29349 raft_consensus.cc:374] T eed1ac06c82147e3bb681f4bcd1d8688 P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:34.362280 29349 raft_consensus.cc:729] T eed1ac06c82147e3bb681f4bcd1d8688 P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8feebcdef3ba428ba32e62016f3ed365, State: Initialized, Role: FOLLOWER
I20240501 20:00:34.362830 29158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4895041b0924276aa587adbb8dc9e98" candidate_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8feebcdef3ba428ba32e62016f3ed365" is_pre_election: true
I20240501 20:00:34.362941 29349 consensus_queue.cc:260] T eed1ac06c82147e3bb681f4bcd1d8688 P 8feebcdef3ba428ba32e62016f3ed365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.364995 29349 ts_tablet_manager.cc:1398] T eed1ac06c82147e3bb681f4bcd1d8688 P 8feebcdef3ba428ba32e62016f3ed365: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240501 20:00:34.365868 29349 tablet_bootstrap.cc:492] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365: Bootstrap starting.
I20240501 20:00:34.366586 29348 tablet_bootstrap.cc:654] T 419e173751fc4cd18b50341160a84d24 P 78bf5e3ff675477cb9f9c874292fa63f: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:34.366456 29232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4895041b0924276aa587adbb8dc9e98" candidate_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78bf5e3ff675477cb9f9c874292fa63f" is_pre_election: true
I20240501 20:00:34.367138 29232 raft_consensus.cc:2454] T f4895041b0924276aa587adbb8dc9e98 P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16f5b5441d8c4a44a9c0a6cca8356822 in term 0.
I20240501 20:00:34.368373 29273 leader_election.cc:302] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16f5b5441d8c4a44a9c0a6cca8356822, 78bf5e3ff675477cb9f9c874292fa63f; no voters:
W20240501 20:00:34.369283 29273 leader_election.cc:341] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193:43937): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:00:34.370242 29369 raft_consensus.cc:2781] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:00:34.370651 29369 raft_consensus.cc:482] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:00:34.372884 29349 tablet_bootstrap.cc:654] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:34.373450 29233 raft_consensus.cc:2454] T 3d51e8d55df849e5ac6e5acdc42ac63c P 78bf5e3ff675477cb9f9c874292fa63f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16f5b5441d8c4a44a9c0a6cca8356822 in term 1.
I20240501 20:00:34.374898 29273 leader_election.cc:302] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16f5b5441d8c4a44a9c0a6cca8356822, 78bf5e3ff675477cb9f9c874292fa63f; no voters: 8feebcdef3ba428ba32e62016f3ed365
I20240501 20:00:34.377214 29354 leader_election.cc:288] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 [CANDIDATE]: Term 1 election: Requested vote from peers 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193:43937), 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194:39263)
I20240501 20:00:34.377485 29372 raft_consensus.cc:2781] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:00:34.381439 29348 tablet_bootstrap.cc:492] T 419e173751fc4cd18b50341160a84d24 P 78bf5e3ff675477cb9f9c874292fa63f: No bootstrap required, opened a new log
I20240501 20:00:34.381330 29372 raft_consensus.cc:686] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 [term 1 LEADER]: Becoming Leader. State: Replica: 16f5b5441d8c4a44a9c0a6cca8356822, State: Running, Role: LEADER
I20240501 20:00:34.381961 29348 ts_tablet_manager.cc:1367] T 419e173751fc4cd18b50341160a84d24 P 78bf5e3ff675477cb9f9c874292fa63f: Time spent bootstrapping tablet: real 0.024s user 0.015s sys 0.002s
I20240501 20:00:34.382560 29372 consensus_queue.cc:237] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.371030 29369 raft_consensus.cc:3037] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:34.384214 29348 raft_consensus.cc:348] T 419e173751fc4cd18b50341160a84d24 P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.384985 29348 raft_consensus.cc:374] T 419e173751fc4cd18b50341160a84d24 P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:34.385315 29348 raft_consensus.cc:729] T 419e173751fc4cd18b50341160a84d24 P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78bf5e3ff675477cb9f9c874292fa63f, State: Initialized, Role: FOLLOWER
I20240501 20:00:34.385963 29348 consensus_queue.cc:260] T 419e173751fc4cd18b50341160a84d24 P 78bf5e3ff675477cb9f9c874292fa63f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.387490 29349 tablet_bootstrap.cc:492] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365: No bootstrap required, opened a new log
I20240501 20:00:34.387928 29349 ts_tablet_manager.cc:1367] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.006s
I20240501 20:00:34.388746 29347 tablet_bootstrap.cc:492] T 27c72ce2560f4d3baf61f0441f5c4129 P 16f5b5441d8c4a44a9c0a6cca8356822: No bootstrap required, opened a new log
I20240501 20:00:34.389288 29347 ts_tablet_manager.cc:1367] T 27c72ce2560f4d3baf61f0441f5c4129 P 16f5b5441d8c4a44a9c0a6cca8356822: Time spent bootstrapping tablet: real 0.077s user 0.018s sys 0.004s
I20240501 20:00:34.390564 29348 ts_tablet_manager.cc:1398] T 419e173751fc4cd18b50341160a84d24 P 78bf5e3ff675477cb9f9c874292fa63f: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20240501 20:00:34.390839 29349 raft_consensus.cc:348] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.391650 29349 raft_consensus.cc:374] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:34.391883 29348 tablet_bootstrap.cc:492] T eed1ac06c82147e3bb681f4bcd1d8688 P 78bf5e3ff675477cb9f9c874292fa63f: Bootstrap starting.
I20240501 20:00:34.391947 29349 raft_consensus.cc:729] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8feebcdef3ba428ba32e62016f3ed365, State: Initialized, Role: FOLLOWER
I20240501 20:00:34.392354 29347 raft_consensus.cc:348] T 27c72ce2560f4d3baf61f0441f5c4129 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.393047 29347 raft_consensus.cc:374] T 27c72ce2560f4d3baf61f0441f5c4129 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:34.393539 29347 raft_consensus.cc:729] T 27c72ce2560f4d3baf61f0441f5c4129 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16f5b5441d8c4a44a9c0a6cca8356822, State: Initialized, Role: FOLLOWER
I20240501 20:00:34.394358 29352 raft_consensus.cc:482] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:34.394248 29347 consensus_queue.cc:260] T 27c72ce2560f4d3baf61f0441f5c4129 P 16f5b5441d8c4a44a9c0a6cca8356822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.394766 29352 raft_consensus.cc:504] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } } peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } }
I20240501 20:00:34.396638 29347 ts_tablet_manager.cc:1398] T 27c72ce2560f4d3baf61f0441f5c4129 P 16f5b5441d8c4a44a9c0a6cca8356822: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240501 20:00:34.397119 29352 leader_election.cc:288] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 78bf5e3ff675477cb9f9c874292fa63f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195:37025), 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193:43937)
I20240501 20:00:34.397516 29347 tablet_bootstrap.cc:492] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 16f5b5441d8c4a44a9c0a6cca8356822: Bootstrap starting.
I20240501 20:00:34.397927 29348 tablet_bootstrap.cc:654] T eed1ac06c82147e3bb681f4bcd1d8688 P 78bf5e3ff675477cb9f9c874292fa63f: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:34.400358 29369 raft_consensus.cc:504] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.404033 29158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4895041b0924276aa587adbb8dc9e98" candidate_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8feebcdef3ba428ba32e62016f3ed365"
W20240501 20:00:34.405809 29273 leader_election.cc:341] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193:43937): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:00:34.409776 29354 raft_consensus.cc:482] T afbcfea6a7894b5ab7c122d3d4134869 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:34.410308 29347 tablet_bootstrap.cc:654] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 16f5b5441d8c4a44a9c0a6cca8356822: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:34.410241 29354 raft_consensus.cc:504] T afbcfea6a7894b5ab7c122d3d4134869 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.412653 29354 leader_election.cc:288] T afbcfea6a7894b5ab7c122d3d4134869 P 16f5b5441d8c4a44a9c0a6cca8356822 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193:43937), 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194:39263)
I20240501 20:00:34.413177 29233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4895041b0924276aa587adbb8dc9e98" candidate_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78bf5e3ff675477cb9f9c874292fa63f"
I20240501 20:00:34.413772 29233 raft_consensus.cc:3037] T f4895041b0924276aa587adbb8dc9e98 P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:34.415553 29158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afbcfea6a7894b5ab7c122d3d4134869" candidate_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8feebcdef3ba428ba32e62016f3ed365" is_pre_election: true
W20240501 20:00:34.417572 29273 leader_election.cc:341] T afbcfea6a7894b5ab7c122d3d4134869 P 16f5b5441d8c4a44a9c0a6cca8356822 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193:43937): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:00:34.417800 29355 raft_consensus.cc:482] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:34.418258 29355 raft_consensus.cc:504] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.420413 29053 catalog_manager.cc:5484] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195). New cstate: current_term: 1 leader_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } health_report { overall_health: HEALTHY } } }
I20240501 20:00:34.423477 29233 raft_consensus.cc:2454] T f4895041b0924276aa587adbb8dc9e98 P 78bf5e3ff675477cb9f9c874292fa63f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16f5b5441d8c4a44a9c0a6cca8356822 in term 1.
I20240501 20:00:34.409432 29369 leader_election.cc:288] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 [CANDIDATE]: Term 1 election: Requested vote from peers 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193:43937), 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194:39263)
I20240501 20:00:34.424700 29232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afbcfea6a7894b5ab7c122d3d4134869" candidate_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78bf5e3ff675477cb9f9c874292fa63f" is_pre_election: true
I20240501 20:00:34.425926 29273 leader_election.cc:302] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16f5b5441d8c4a44a9c0a6cca8356822, 78bf5e3ff675477cb9f9c874292fa63f; no voters: 8feebcdef3ba428ba32e62016f3ed365
I20240501 20:00:34.426008 29232 raft_consensus.cc:2454] T afbcfea6a7894b5ab7c122d3d4134869 P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16f5b5441d8c4a44a9c0a6cca8356822 in term 0.
I20240501 20:00:34.426965 29372 raft_consensus.cc:2781] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:00:34.427459 29372 raft_consensus.cc:686] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 1 LEADER]: Becoming Leader. State: Replica: 16f5b5441d8c4a44a9c0a6cca8356822, State: Running, Role: LEADER
I20240501 20:00:34.427405 29273 leader_election.cc:302] T afbcfea6a7894b5ab7c122d3d4134869 P 16f5b5441d8c4a44a9c0a6cca8356822 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16f5b5441d8c4a44a9c0a6cca8356822, 78bf5e3ff675477cb9f9c874292fa63f; no voters: 8feebcdef3ba428ba32e62016f3ed365
I20240501 20:00:34.428591 29373 raft_consensus.cc:2781] T afbcfea6a7894b5ab7c122d3d4134869 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:00:34.428403 29372 consensus_queue.cc:237] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.428980 29373 raft_consensus.cc:482] T afbcfea6a7894b5ab7c122d3d4134869 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:00:34.428896 29349 consensus_queue.cc:260] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.429432 29373 raft_consensus.cc:3037] T afbcfea6a7894b5ab7c122d3d4134869 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:34.434712 29355 leader_election.cc:288] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194:39263), 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195:37025)
W20240501 20:00:34.442309 29104 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240501 20:00:34.442731 29373 raft_consensus.cc:504] T afbcfea6a7894b5ab7c122d3d4134869 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.443152 29352 raft_consensus.cc:482] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:34.443760 29352 raft_consensus.cc:504] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.446121 29352 leader_election.cc:288] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193:43937), 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195:37025)
I20240501 20:00:34.455157 29349 ts_tablet_manager.cc:1398] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365: Time spent starting tablet: real 0.067s user 0.004s sys 0.014s
I20240501 20:00:34.456600 29349 tablet_bootstrap.cc:492] T 27c72ce2560f4d3baf61f0441f5c4129 P 8feebcdef3ba428ba32e62016f3ed365: Bootstrap starting.
I20240501 20:00:34.462553 29307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d9b61e9c98e4d19b9272fe70fab0c3d" candidate_uuid: "78bf5e3ff675477cb9f9c874292fa63f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" is_pre_election: true
I20240501 20:00:34.463541 29307 raft_consensus.cc:2454] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78bf5e3ff675477cb9f9c874292fa63f in term 0.
I20240501 20:00:34.464027 29349 tablet_bootstrap.cc:654] T 27c72ce2560f4d3baf61f0441f5c4129 P 8feebcdef3ba428ba32e62016f3ed365: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:34.464934 29306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bd59b3e54144bd2986538a355cadd9b" candidate_uuid: "78bf5e3ff675477cb9f9c874292fa63f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" is_pre_election: true
I20240501 20:00:34.466501 29347 tablet_bootstrap.cc:492] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 16f5b5441d8c4a44a9c0a6cca8356822: No bootstrap required, opened a new log
I20240501 20:00:34.467003 29347 ts_tablet_manager.cc:1367] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 16f5b5441d8c4a44a9c0a6cca8356822: Time spent bootstrapping tablet: real 0.070s user 0.012s sys 0.009s
I20240501 20:00:34.466532 29197 leader_election.cc:302] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 78bf5e3ff675477cb9f9c874292fa63f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16f5b5441d8c4a44a9c0a6cca8356822, 78bf5e3ff675477cb9f9c874292fa63f; no voters:
I20240501 20:00:34.469043 29352 raft_consensus.cc:2781] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:00:34.469532 29348 tablet_bootstrap.cc:492] T eed1ac06c82147e3bb681f4bcd1d8688 P 78bf5e3ff675477cb9f9c874292fa63f: No bootstrap required, opened a new log
I20240501 20:00:34.469635 29352 raft_consensus.cc:482] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:00:34.470018 29348 ts_tablet_manager.cc:1367] T eed1ac06c82147e3bb681f4bcd1d8688 P 78bf5e3ff675477cb9f9c874292fa63f: Time spent bootstrapping tablet: real 0.078s user 0.012s sys 0.012s
I20240501 20:00:34.470137 29352 raft_consensus.cc:3037] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:34.471632 29347 raft_consensus.cc:348] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.472586 29347 raft_consensus.cc:374] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:34.473013 29347 raft_consensus.cc:729] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16f5b5441d8c4a44a9c0a6cca8356822, State: Initialized, Role: FOLLOWER
W20240501 20:00:34.469157 29197 leader_election.cc:341] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195:37025): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:00:34.474068 29347 consensus_queue.cc:260] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 16f5b5441d8c4a44a9c0a6cca8356822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.476811 29347 ts_tablet_manager.cc:1398] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 16f5b5441d8c4a44a9c0a6cca8356822: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20240501 20:00:34.476935 29349 tablet_bootstrap.cc:492] T 27c72ce2560f4d3baf61f0441f5c4129 P 8feebcdef3ba428ba32e62016f3ed365: No bootstrap required, opened a new log
I20240501 20:00:34.476852 29158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afbcfea6a7894b5ab7c122d3d4134869" candidate_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8feebcdef3ba428ba32e62016f3ed365"
I20240501 20:00:34.477553 29349 ts_tablet_manager.cc:1367] T 27c72ce2560f4d3baf61f0441f5c4129 P 8feebcdef3ba428ba32e62016f3ed365: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.009s
I20240501 20:00:34.478047 29347 tablet_bootstrap.cc:492] T 5bd59b3e54144bd2986538a355cadd9b P 16f5b5441d8c4a44a9c0a6cca8356822: Bootstrap starting.
W20240501 20:00:34.478421 29273 leader_election.cc:341] T afbcfea6a7894b5ab7c122d3d4134869 P 16f5b5441d8c4a44a9c0a6cca8356822 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193:43937): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:00:34.478920 29352 raft_consensus.cc:504] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 78bf5e3ff675477cb9f9c874292fa63f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } } peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } }
I20240501 20:00:34.479801 29373 leader_election.cc:288] T afbcfea6a7894b5ab7c122d3d4134869 P 16f5b5441d8c4a44a9c0a6cca8356822 [CANDIDATE]: Term 1 election: Requested vote from peers 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193:43937), 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194:39263)
I20240501 20:00:34.480429 29232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afbcfea6a7894b5ab7c122d3d4134869" candidate_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78bf5e3ff675477cb9f9c874292fa63f"
I20240501 20:00:34.480306 29349 raft_consensus.cc:348] T 27c72ce2560f4d3baf61f0441f5c4129 P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.481006 29232 raft_consensus.cc:3037] T afbcfea6a7894b5ab7c122d3d4134869 P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:34.481047 29349 raft_consensus.cc:374] T 27c72ce2560f4d3baf61f0441f5c4129 P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:34.481511 29349 raft_consensus.cc:729] T 27c72ce2560f4d3baf61f0441f5c4129 P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8feebcdef3ba428ba32e62016f3ed365, State: Initialized, Role: FOLLOWER
I20240501 20:00:34.482335 29349 consensus_queue.cc:260] T 27c72ce2560f4d3baf61f0441f5c4129 P 8feebcdef3ba428ba32e62016f3ed365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.484589 29349 ts_tablet_manager.cc:1398] T 27c72ce2560f4d3baf61f0441f5c4129 P 8feebcdef3ba428ba32e62016f3ed365: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240501 20:00:34.485462 29349 tablet_bootstrap.cc:492] T afbcfea6a7894b5ab7c122d3d4134869 P 8feebcdef3ba428ba32e62016f3ed365: Bootstrap starting.
I20240501 20:00:34.487649 29158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d9b61e9c98e4d19b9272fe70fab0c3d" candidate_uuid: "78bf5e3ff675477cb9f9c874292fa63f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8feebcdef3ba428ba32e62016f3ed365" is_pre_election: true
I20240501 20:00:34.489491 29158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bd59b3e54144bd2986538a355cadd9b" candidate_uuid: "78bf5e3ff675477cb9f9c874292fa63f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8feebcdef3ba428ba32e62016f3ed365" is_pre_election: true
I20240501 20:00:34.490271 29158 raft_consensus.cc:2454] T 5bd59b3e54144bd2986538a355cadd9b P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78bf5e3ff675477cb9f9c874292fa63f in term 0.
W20240501 20:00:34.493790 29199 leader_election.cc:341] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 78bf5e3ff675477cb9f9c874292fa63f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193:43937): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:00:34.495034 29199 leader_election.cc:302] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78bf5e3ff675477cb9f9c874292fa63f, 8feebcdef3ba428ba32e62016f3ed365; no voters: 16f5b5441d8c4a44a9c0a6cca8356822
I20240501 20:00:34.499799 29232 raft_consensus.cc:2454] T afbcfea6a7894b5ab7c122d3d4134869 P 78bf5e3ff675477cb9f9c874292fa63f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16f5b5441d8c4a44a9c0a6cca8356822 in term 1.
I20240501 20:00:34.502195 29382 raft_consensus.cc:482] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:34.503068 29347 tablet_bootstrap.cc:654] T 5bd59b3e54144bd2986538a355cadd9b P 16f5b5441d8c4a44a9c0a6cca8356822: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:34.502712 29382 raft_consensus.cc:504] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.505349 29382 leader_election.cc:288] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194:39263), 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195:37025)
I20240501 20:00:34.507459 29348 raft_consensus.cc:348] T eed1ac06c82147e3bb681f4bcd1d8688 P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.508404 29348 raft_consensus.cc:374] T eed1ac06c82147e3bb681f4bcd1d8688 P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:34.508754 29348 raft_consensus.cc:729] T eed1ac06c82147e3bb681f4bcd1d8688 P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78bf5e3ff675477cb9f9c874292fa63f, State: Initialized, Role: FOLLOWER
I20240501 20:00:34.509531 29348 consensus_queue.cc:260] T eed1ac06c82147e3bb681f4bcd1d8688 P 78bf5e3ff675477cb9f9c874292fa63f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.510707 29349 tablet_bootstrap.cc:654] T afbcfea6a7894b5ab7c122d3d4134869 P 8feebcdef3ba428ba32e62016f3ed365: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:34.518482 29306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d9b61e9c98e4d19b9272fe70fab0c3d" candidate_uuid: "78bf5e3ff675477cb9f9c874292fa63f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16f5b5441d8c4a44a9c0a6cca8356822"
I20240501 20:00:34.519340 29306 raft_consensus.cc:3037] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:34.522807 29349 tablet_bootstrap.cc:492] T afbcfea6a7894b5ab7c122d3d4134869 P 8feebcdef3ba428ba32e62016f3ed365: No bootstrap required, opened a new log
I20240501 20:00:34.523314 29349 ts_tablet_manager.cc:1367] T afbcfea6a7894b5ab7c122d3d4134869 P 8feebcdef3ba428ba32e62016f3ed365: Time spent bootstrapping tablet: real 0.038s user 0.010s sys 0.020s
I20240501 20:00:34.525964 29349 raft_consensus.cc:348] T afbcfea6a7894b5ab7c122d3d4134869 P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.526656 29349 raft_consensus.cc:374] T afbcfea6a7894b5ab7c122d3d4134869 P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:34.526969 29349 raft_consensus.cc:729] T afbcfea6a7894b5ab7c122d3d4134869 P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8feebcdef3ba428ba32e62016f3ed365, State: Initialized, Role: FOLLOWER
I20240501 20:00:34.527220 29306 raft_consensus.cc:2454] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 16f5b5441d8c4a44a9c0a6cca8356822 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78bf5e3ff675477cb9f9c874292fa63f in term 1.
I20240501 20:00:34.527854 29349 consensus_queue.cc:260] T afbcfea6a7894b5ab7c122d3d4134869 P 8feebcdef3ba428ba32e62016f3ed365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.528393 29197 leader_election.cc:302] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 78bf5e3ff675477cb9f9c874292fa63f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16f5b5441d8c4a44a9c0a6cca8356822, 78bf5e3ff675477cb9f9c874292fa63f; no voters:
I20240501 20:00:34.533205 29349 ts_tablet_manager.cc:1398] T afbcfea6a7894b5ab7c122d3d4134869 P 8feebcdef3ba428ba32e62016f3ed365: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20240501 20:00:34.534358 29349 tablet_bootstrap.cc:492] T f4895041b0924276aa587adbb8dc9e98 P 8feebcdef3ba428ba32e62016f3ed365: Bootstrap starting.
I20240501 20:00:34.540680 29352 leader_election.cc:288] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 78bf5e3ff675477cb9f9c874292fa63f [CANDIDATE]: Term 1 election: Requested vote from peers 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195:37025), 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193:43937)
I20240501 20:00:34.541316 29352 raft_consensus.cc:2781] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:00:34.541707 29352 raft_consensus.cc:482] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:00:34.541828 29349 tablet_bootstrap.cc:654] T f4895041b0924276aa587adbb8dc9e98 P 8feebcdef3ba428ba32e62016f3ed365: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:34.542084 29352 raft_consensus.cc:3037] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:34.547927 29391 raft_consensus.cc:2781] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 78bf5e3ff675477cb9f9c874292fa63f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:00:34.548617 29391 raft_consensus.cc:686] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 78bf5e3ff675477cb9f9c874292fa63f [term 1 LEADER]: Becoming Leader. State: Replica: 78bf5e3ff675477cb9f9c874292fa63f, State: Running, Role: LEADER
I20240501 20:00:34.550209 29382 raft_consensus.cc:482] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:34.550668 29382 raft_consensus.cc:504] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.552594 29382 leader_election.cc:288] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194:39263), 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195:37025)
I20240501 20:00:34.556921 29349 tablet_bootstrap.cc:492] T f4895041b0924276aa587adbb8dc9e98 P 8feebcdef3ba428ba32e62016f3ed365: No bootstrap required, opened a new log
I20240501 20:00:34.557425 29349 ts_tablet_manager.cc:1367] T f4895041b0924276aa587adbb8dc9e98 P 8feebcdef3ba428ba32e62016f3ed365: Time spent bootstrapping tablet: real 0.023s user 0.017s sys 0.000s
W20240501 20:00:34.559177 29199 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:43937, user_credentials={real_user=slave}} blocked reactor thread for 64227.8us
I20240501 20:00:34.559607 29232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "419e173751fc4cd18b50341160a84d24" candidate_uuid: "8feebcdef3ba428ba32e62016f3ed365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78bf5e3ff675477cb9f9c874292fa63f" is_pre_election: true
I20240501 20:00:34.559983 29349 raft_consensus.cc:348] T f4895041b0924276aa587adbb8dc9e98 P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.560353 29273 leader_election.cc:302] T afbcfea6a7894b5ab7c122d3d4134869 P 16f5b5441d8c4a44a9c0a6cca8356822 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16f5b5441d8c4a44a9c0a6cca8356822, 78bf5e3ff675477cb9f9c874292fa63f; no voters: 8feebcdef3ba428ba32e62016f3ed365
I20240501 20:00:34.560515 29232 raft_consensus.cc:2454] T 419e173751fc4cd18b50341160a84d24 P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8feebcdef3ba428ba32e62016f3ed365 in term 0.
I20240501 20:00:34.561462 29372 raft_consensus.cc:2781] T afbcfea6a7894b5ab7c122d3d4134869 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:00:34.561964 29372 raft_consensus.cc:686] T afbcfea6a7894b5ab7c122d3d4134869 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 1 LEADER]: Becoming Leader. State: Replica: 16f5b5441d8c4a44a9c0a6cca8356822, State: Running, Role: LEADER
I20240501 20:00:34.561800 29233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d057ab88d0084efe8d2fe408a42a24b7" candidate_uuid: "8feebcdef3ba428ba32e62016f3ed365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78bf5e3ff675477cb9f9c874292fa63f" is_pre_election: true
I20240501 20:00:34.562662 29233 raft_consensus.cc:2454] T d057ab88d0084efe8d2fe408a42a24b7 P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8feebcdef3ba428ba32e62016f3ed365 in term 0.
I20240501 20:00:34.549707 29391 consensus_queue.cc:237] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 78bf5e3ff675477cb9f9c874292fa63f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } } peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } }
I20240501 20:00:34.562846 29372 consensus_queue.cc:237] T afbcfea6a7894b5ab7c122d3d4134869 P 16f5b5441d8c4a44a9c0a6cca8356822 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.560719 29349 raft_consensus.cc:374] T f4895041b0924276aa587adbb8dc9e98 P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:34.564328 29349 raft_consensus.cc:729] T f4895041b0924276aa587adbb8dc9e98 P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8feebcdef3ba428ba32e62016f3ed365, State: Initialized, Role: FOLLOWER
I20240501 20:00:34.564642 29233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c7b0c234f7d4cc0a0848e47e24c5d8b" candidate_uuid: "8feebcdef3ba428ba32e62016f3ed365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78bf5e3ff675477cb9f9c874292fa63f" is_pre_election: true
I20240501 20:00:34.565742 29349 consensus_queue.cc:260] T f4895041b0924276aa587adbb8dc9e98 P 8feebcdef3ba428ba32e62016f3ed365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.566601 29158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d9b61e9c98e4d19b9272fe70fab0c3d" candidate_uuid: "78bf5e3ff675477cb9f9c874292fa63f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8feebcdef3ba428ba32e62016f3ed365"
I20240501 20:00:34.567880 29373 raft_consensus.cc:482] T 27c72ce2560f4d3baf61f0441f5c4129 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:34.568426 29373 raft_consensus.cc:504] T 27c72ce2560f4d3baf61f0441f5c4129 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.568671 29124 leader_election.cc:302] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78bf5e3ff675477cb9f9c874292fa63f, 8feebcdef3ba428ba32e62016f3ed365; no voters:
I20240501 20:00:34.569778 29382 raft_consensus.cc:2781] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:00:34.570236 29382 raft_consensus.cc:482] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:00:34.570662 29382 raft_consensus.cc:3037] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:34.570974 29158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c72ce2560f4d3baf61f0441f5c4129" candidate_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8feebcdef3ba428ba32e62016f3ed365" is_pre_election: true
I20240501 20:00:34.571578 29158 raft_consensus.cc:2454] T 27c72ce2560f4d3baf61f0441f5c4129 P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16f5b5441d8c4a44a9c0a6cca8356822 in term 0.
I20240501 20:00:34.571738 29349 ts_tablet_manager.cc:1398] T f4895041b0924276aa587adbb8dc9e98 P 8feebcdef3ba428ba32e62016f3ed365: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20240501 20:00:34.572645 29349 tablet_bootstrap.cc:492] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 8feebcdef3ba428ba32e62016f3ed365: Bootstrap starting.
I20240501 20:00:34.572728 29273 leader_election.cc:302] T 27c72ce2560f4d3baf61f0441f5c4129 P 16f5b5441d8c4a44a9c0a6cca8356822 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16f5b5441d8c4a44a9c0a6cca8356822, 8feebcdef3ba428ba32e62016f3ed365; no voters:
I20240501 20:00:34.573546 29354 raft_consensus.cc:2781] T 27c72ce2560f4d3baf61f0441f5c4129 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:00:34.573904 29354 raft_consensus.cc:482] T 27c72ce2560f4d3baf61f0441f5c4129 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:00:34.574224 29354 raft_consensus.cc:3037] T 27c72ce2560f4d3baf61f0441f5c4129 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:34.580922 29373 leader_election.cc:288] T 27c72ce2560f4d3baf61f0441f5c4129 P 16f5b5441d8c4a44a9c0a6cca8356822 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193:43937), 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194:39263)
I20240501 20:00:34.581897 29124 leader_election.cc:302] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78bf5e3ff675477cb9f9c874292fa63f, 8feebcdef3ba428ba32e62016f3ed365; no voters:
I20240501 20:00:34.583092 29355 raft_consensus.cc:2781] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:00:34.583652 29355 raft_consensus.cc:482] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:00:34.583153 29354 raft_consensus.cc:504] T 27c72ce2560f4d3baf61f0441f5c4129 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.583982 29355 raft_consensus.cc:3037] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:34.586030 29349 tablet_bootstrap.cc:654] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 8feebcdef3ba428ba32e62016f3ed365: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:34.586076 29158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c72ce2560f4d3baf61f0441f5c4129" candidate_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8feebcdef3ba428ba32e62016f3ed365"
I20240501 20:00:34.586764 29158 raft_consensus.cc:3037] T 27c72ce2560f4d3baf61f0441f5c4129 P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:34.593091 29233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c72ce2560f4d3baf61f0441f5c4129" candidate_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78bf5e3ff675477cb9f9c874292fa63f" is_pre_election: true
W20240501 20:00:34.594589 29199 leader_election.cc:341] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 78bf5e3ff675477cb9f9c874292fa63f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193:43937): Illegal state: must be running to vote when last-logged opid is not known
W20240501 20:00:34.595779 29273 leader_election.cc:341] T 27c72ce2560f4d3baf61f0441f5c4129 P 16f5b5441d8c4a44a9c0a6cca8356822 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194:39263): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:00:34.597338 29349 tablet_bootstrap.cc:492] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 8feebcdef3ba428ba32e62016f3ed365: No bootstrap required, opened a new log
I20240501 20:00:34.597764 29349 ts_tablet_manager.cc:1367] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 8feebcdef3ba428ba32e62016f3ed365: Time spent bootstrapping tablet: real 0.025s user 0.015s sys 0.000s
I20240501 20:00:34.597891 29348 ts_tablet_manager.cc:1398] T eed1ac06c82147e3bb681f4bcd1d8688 P 78bf5e3ff675477cb9f9c874292fa63f: Time spent starting tablet: real 0.128s user 0.009s sys 0.000s
I20240501 20:00:34.599102 29348 tablet_bootstrap.cc:492] T 27c72ce2560f4d3baf61f0441f5c4129 P 78bf5e3ff675477cb9f9c874292fa63f: Bootstrap starting.
I20240501 20:00:34.599601 29158 raft_consensus.cc:2454] T 27c72ce2560f4d3baf61f0441f5c4129 P 8feebcdef3ba428ba32e62016f3ed365 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16f5b5441d8c4a44a9c0a6cca8356822 in term 1.
I20240501 20:00:34.600193 29349 raft_consensus.cc:348] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } } peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } }
I20240501 20:00:34.600889 29354 leader_election.cc:288] T 27c72ce2560f4d3baf61f0441f5c4129 P 16f5b5441d8c4a44a9c0a6cca8356822 [CANDIDATE]: Term 1 election: Requested vote from peers 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193:43937), 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194:39263)
I20240501 20:00:34.600893 29349 raft_consensus.cc:374] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:34.601367 29349 raft_consensus.cc:729] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8feebcdef3ba428ba32e62016f3ed365, State: Initialized, Role: FOLLOWER
I20240501 20:00:34.602002 29349 consensus_queue.cc:260] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 8feebcdef3ba428ba32e62016f3ed365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } } peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } }
I20240501 20:00:34.603202 29233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c72ce2560f4d3baf61f0441f5c4129" candidate_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78bf5e3ff675477cb9f9c874292fa63f"
I20240501 20:00:34.604194 29355 raft_consensus.cc:504] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
W20240501 20:00:34.604663 29273 leader_election.cc:341] T 27c72ce2560f4d3baf61f0441f5c4129 P 16f5b5441d8c4a44a9c0a6cca8356822 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194:39263): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:00:34.608976 29306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "419e173751fc4cd18b50341160a84d24" candidate_uuid: "8feebcdef3ba428ba32e62016f3ed365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" is_pre_election: true
I20240501 20:00:34.609616 29307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d057ab88d0084efe8d2fe408a42a24b7" candidate_uuid: "8feebcdef3ba428ba32e62016f3ed365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" is_pre_election: true
I20240501 20:00:34.610230 29398 raft_consensus.cc:482] T f4895041b0924276aa587adbb8dc9e98 P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:34.611006 29273 leader_election.cc:302] T 27c72ce2560f4d3baf61f0441f5c4129 P 16f5b5441d8c4a44a9c0a6cca8356822 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16f5b5441d8c4a44a9c0a6cca8356822, 8feebcdef3ba428ba32e62016f3ed365; no voters: 78bf5e3ff675477cb9f9c874292fa63f
I20240501 20:00:34.612159 29372 raft_consensus.cc:2781] T 27c72ce2560f4d3baf61f0441f5c4129 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:00:34.612234 29307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c7b0c234f7d4cc0a0848e47e24c5d8b" candidate_uuid: "8feebcdef3ba428ba32e62016f3ed365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" is_pre_election: true
I20240501 20:00:34.612710 29372 raft_consensus.cc:686] T 27c72ce2560f4d3baf61f0441f5c4129 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 1 LEADER]: Becoming Leader. State: Replica: 16f5b5441d8c4a44a9c0a6cca8356822, State: Running, Role: LEADER
I20240501 20:00:34.613077 29307 raft_consensus.cc:2454] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8feebcdef3ba428ba32e62016f3ed365 in term 0.
I20240501 20:00:34.613546 29372 consensus_queue.cc:237] T 27c72ce2560f4d3baf61f0441f5c4129 P 16f5b5441d8c4a44a9c0a6cca8356822 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.616505 29349 ts_tablet_manager.cc:1398] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 8feebcdef3ba428ba32e62016f3ed365: Time spent starting tablet: real 0.018s user 0.010s sys 0.004s
I20240501 20:00:34.617166 29382 raft_consensus.cc:504] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
W20240501 20:00:34.614761 29122 leader_election.cc:341] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195:37025): Illegal state: must be running to vote when last-logged opid is not known
W20240501 20:00:34.618477 29122 leader_election.cc:341] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195:37025): Illegal state: must be running to vote when last-logged opid is not known
W20240501 20:00:34.619079 29124 leader_election.cc:341] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194:39263): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:00:34.617529 29349 tablet_bootstrap.cc:492] T 3d51e8d55df849e5ac6e5acdc42ac63c P 8feebcdef3ba428ba32e62016f3ed365: Bootstrap starting.
I20240501 20:00:34.619738 29347 tablet_bootstrap.cc:492] T 5bd59b3e54144bd2986538a355cadd9b P 16f5b5441d8c4a44a9c0a6cca8356822: No bootstrap required, opened a new log
I20240501 20:00:34.620298 29347 ts_tablet_manager.cc:1367] T 5bd59b3e54144bd2986538a355cadd9b P 16f5b5441d8c4a44a9c0a6cca8356822: Time spent bootstrapping tablet: real 0.143s user 0.057s sys 0.065s
I20240501 20:00:34.623008 29347 raft_consensus.cc:348] T 5bd59b3e54144bd2986538a355cadd9b P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.623808 29347 raft_consensus.cc:374] T 5bd59b3e54144bd2986538a355cadd9b P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:34.624187 29347 raft_consensus.cc:729] T 5bd59b3e54144bd2986538a355cadd9b P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16f5b5441d8c4a44a9c0a6cca8356822, State: Initialized, Role: FOLLOWER
I20240501 20:00:34.624866 29349 tablet_bootstrap.cc:654] T 3d51e8d55df849e5ac6e5acdc42ac63c P 8feebcdef3ba428ba32e62016f3ed365: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:34.624656 29056 catalog_manager.cc:5484] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 78bf5e3ff675477cb9f9c874292fa63f reported cstate change: term changed from 0 to 1, leader changed from <none> to 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194). New cstate: current_term: 1 leader_uuid: "78bf5e3ff675477cb9f9c874292fa63f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } health_report { overall_health: HEALTHY } } }
I20240501 20:00:34.625118 29347 consensus_queue.cc:260] T 5bd59b3e54144bd2986538a355cadd9b P 16f5b5441d8c4a44a9c0a6cca8356822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.626470 29355 leader_election.cc:288] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 1 election: Requested vote from peers 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194:39263), 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195:37025)
I20240501 20:00:34.625548 29232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d057ab88d0084efe8d2fe408a42a24b7" candidate_uuid: "8feebcdef3ba428ba32e62016f3ed365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78bf5e3ff675477cb9f9c874292fa63f"
I20240501 20:00:34.627056 29232 raft_consensus.cc:3037] T d057ab88d0084efe8d2fe408a42a24b7 P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:34.610720 29398 raft_consensus.cc:504] T f4895041b0924276aa587adbb8dc9e98 P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.629102 29347 ts_tablet_manager.cc:1398] T 5bd59b3e54144bd2986538a355cadd9b P 16f5b5441d8c4a44a9c0a6cca8356822: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240501 20:00:34.630095 29347 tablet_bootstrap.cc:492] T 419e173751fc4cd18b50341160a84d24 P 16f5b5441d8c4a44a9c0a6cca8356822: Bootstrap starting.
I20240501 20:00:34.630448 29233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "419e173751fc4cd18b50341160a84d24" candidate_uuid: "8feebcdef3ba428ba32e62016f3ed365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78bf5e3ff675477cb9f9c874292fa63f"
I20240501 20:00:34.630385 29231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4895041b0924276aa587adbb8dc9e98" candidate_uuid: "8feebcdef3ba428ba32e62016f3ed365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78bf5e3ff675477cb9f9c874292fa63f" is_pre_election: true
I20240501 20:00:34.631382 29231 raft_consensus.cc:2379] T f4895041b0924276aa587adbb8dc9e98 P 78bf5e3ff675477cb9f9c874292fa63f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8feebcdef3ba428ba32e62016f3ed365 in current term 1: Already voted for candidate 16f5b5441d8c4a44a9c0a6cca8356822 in this term.
I20240501 20:00:34.631659 29348 tablet_bootstrap.cc:654] T 27c72ce2560f4d3baf61f0441f5c4129 P 78bf5e3ff675477cb9f9c874292fa63f: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:34.631901 29122 leader_election.cc:302] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16f5b5441d8c4a44a9c0a6cca8356822, 8feebcdef3ba428ba32e62016f3ed365; no voters: 78bf5e3ff675477cb9f9c874292fa63f
I20240501 20:00:34.633211 29382 leader_election.cc:288] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 1 election: Requested vote from peers 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194:39263), 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195:37025)
I20240501 20:00:34.633821 29382 raft_consensus.cc:2781] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:00:34.633939 29398 leader_election.cc:288] T f4895041b0924276aa587adbb8dc9e98 P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194:39263), 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195:37025)
I20240501 20:00:34.634207 29382 raft_consensus.cc:482] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:00:34.634312 29307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d057ab88d0084efe8d2fe408a42a24b7" candidate_uuid: "8feebcdef3ba428ba32e62016f3ed365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16f5b5441d8c4a44a9c0a6cca8356822"
I20240501 20:00:34.634666 29382 raft_consensus.cc:3037] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:34.636890 29347 tablet_bootstrap.cc:654] T 419e173751fc4cd18b50341160a84d24 P 16f5b5441d8c4a44a9c0a6cca8356822: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:34.640170 29307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "419e173751fc4cd18b50341160a84d24" candidate_uuid: "8feebcdef3ba428ba32e62016f3ed365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16f5b5441d8c4a44a9c0a6cca8356822"
W20240501 20:00:34.642153 29122 leader_election.cc:341] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195:37025): Illegal state: must be running to vote when last-logged opid is not known
W20240501 20:00:34.643364 29122 leader_election.cc:341] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195:37025): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:00:34.644621 29307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4895041b0924276aa587adbb8dc9e98" candidate_uuid: "8feebcdef3ba428ba32e62016f3ed365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" is_pre_election: true
I20240501 20:00:34.632159 29233 raft_consensus.cc:3037] T 419e173751fc4cd18b50341160a84d24 P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:34.645211 29382 raft_consensus.cc:504] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 8feebcdef3ba428ba32e62016f3ed365 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.647675 29124 leader_election.cc:302] T f4895041b0924276aa587adbb8dc9e98 P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8feebcdef3ba428ba32e62016f3ed365; no voters: 16f5b5441d8c4a44a9c0a6cca8356822, 78bf5e3ff675477cb9f9c874292fa63f
I20240501 20:00:34.649174 29355 raft_consensus.cc:3037] T f4895041b0924276aa587adbb8dc9e98 P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:34.649106 29307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c7b0c234f7d4cc0a0848e47e24c5d8b" candidate_uuid: "8feebcdef3ba428ba32e62016f3ed365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16f5b5441d8c4a44a9c0a6cca8356822"
I20240501 20:00:34.649948 29307 raft_consensus.cc:3037] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:34.652442 29347 tablet_bootstrap.cc:492] T 419e173751fc4cd18b50341160a84d24 P 16f5b5441d8c4a44a9c0a6cca8356822: No bootstrap required, opened a new log
I20240501 20:00:34.653005 29347 ts_tablet_manager.cc:1367] T 419e173751fc4cd18b50341160a84d24 P 16f5b5441d8c4a44a9c0a6cca8356822: Time spent bootstrapping tablet: real 0.023s user 0.017s sys 0.000s
I20240501 20:00:34.653720 29398 raft_consensus.cc:482] T eed1ac06c82147e3bb681f4bcd1d8688 P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:34.654184 29398 raft_consensus.cc:504] T eed1ac06c82147e3bb681f4bcd1d8688 P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.655946 29355 raft_consensus.cc:2726] T f4895041b0924276aa587adbb8dc9e98 P 8feebcdef3ba428ba32e62016f3ed365 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:00:34.656525 29398 leader_election.cc:288] T eed1ac06c82147e3bb681f4bcd1d8688 P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194:39263), 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195:37025)
I20240501 20:00:34.656795 29307 raft_consensus.cc:2454] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 16f5b5441d8c4a44a9c0a6cca8356822 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8feebcdef3ba428ba32e62016f3ed365 in term 1.
I20240501 20:00:34.656740 29231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c7b0c234f7d4cc0a0848e47e24c5d8b" candidate_uuid: "8feebcdef3ba428ba32e62016f3ed365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78bf5e3ff675477cb9f9c874292fa63f"
I20240501 20:00:34.655912 29347 raft_consensus.cc:348] T 419e173751fc4cd18b50341160a84d24 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.657739 29347 raft_consensus.cc:374] T 419e173751fc4cd18b50341160a84d24 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240501 20:00:34.658121 29124 leader_election.cc:341] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194:39263): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:00:34.658185 29347 raft_consensus.cc:729] T 419e173751fc4cd18b50341160a84d24 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16f5b5441d8c4a44a9c0a6cca8356822, State: Initialized, Role: FOLLOWER
I20240501 20:00:34.659526 29231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eed1ac06c82147e3bb681f4bcd1d8688" candidate_uuid: "8feebcdef3ba428ba32e62016f3ed365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78bf5e3ff675477cb9f9c874292fa63f" is_pre_election: true
I20240501 20:00:34.659269 29347 consensus_queue.cc:260] T 419e173751fc4cd18b50341160a84d24 P 16f5b5441d8c4a44a9c0a6cca8356822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.660202 29231 raft_consensus.cc:2454] T eed1ac06c82147e3bb681f4bcd1d8688 P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8feebcdef3ba428ba32e62016f3ed365 in term 0.
I20240501 20:00:34.661530 29124 leader_election.cc:302] T eed1ac06c82147e3bb681f4bcd1d8688 P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78bf5e3ff675477cb9f9c874292fa63f, 8feebcdef3ba428ba32e62016f3ed365; no voters:
I20240501 20:00:34.662473 29355 raft_consensus.cc:2781] T eed1ac06c82147e3bb681f4bcd1d8688 P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:00:34.661487 29122 leader_election.cc:302] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16f5b5441d8c4a44a9c0a6cca8356822, 8feebcdef3ba428ba32e62016f3ed365; no voters: 78bf5e3ff675477cb9f9c874292fa63f
I20240501 20:00:34.659551 29382 leader_election.cc:288] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 1 election: Requested vote from peers 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194:39263), 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195:37025)
I20240501 20:00:34.662845 29355 raft_consensus.cc:482] T eed1ac06c82147e3bb681f4bcd1d8688 P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:00:34.663275 29355 raft_consensus.cc:3037] T eed1ac06c82147e3bb681f4bcd1d8688 P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:34.663350 29348 tablet_bootstrap.cc:492] T 27c72ce2560f4d3baf61f0441f5c4129 P 78bf5e3ff675477cb9f9c874292fa63f: No bootstrap required, opened a new log
I20240501 20:00:34.663900 29348 ts_tablet_manager.cc:1367] T 27c72ce2560f4d3baf61f0441f5c4129 P 78bf5e3ff675477cb9f9c874292fa63f: Time spent bootstrapping tablet: real 0.065s user 0.011s sys 0.003s
I20240501 20:00:34.666118 29398 raft_consensus.cc:2781] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 8feebcdef3ba428ba32e62016f3ed365 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:00:34.666595 29398 raft_consensus.cc:686] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 8feebcdef3ba428ba32e62016f3ed365 [term 1 LEADER]: Becoming Leader. State: Replica: 8feebcdef3ba428ba32e62016f3ed365, State: Running, Role: LEADER
I20240501 20:00:34.667059 29307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eed1ac06c82147e3bb681f4bcd1d8688" candidate_uuid: "8feebcdef3ba428ba32e62016f3ed365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" is_pre_election: true
I20240501 20:00:34.667436 29398 consensus_queue.cc:237] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 8feebcdef3ba428ba32e62016f3ed365 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
W20240501 20:00:34.668398 29122 leader_election.cc:341] T eed1ac06c82147e3bb681f4bcd1d8688 P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195:37025): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:00:34.666436 29348 raft_consensus.cc:348] T 27c72ce2560f4d3baf61f0441f5c4129 P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.669767 29348 raft_consensus.cc:374] T 27c72ce2560f4d3baf61f0441f5c4129 P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:34.670039 29348 raft_consensus.cc:729] T 27c72ce2560f4d3baf61f0441f5c4129 P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78bf5e3ff675477cb9f9c874292fa63f, State: Initialized, Role: FOLLOWER
I20240501 20:00:34.662453 29347 ts_tablet_manager.cc:1398] T 419e173751fc4cd18b50341160a84d24 P 16f5b5441d8c4a44a9c0a6cca8356822: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20240501 20:00:34.670733 29348 consensus_queue.cc:260] T 27c72ce2560f4d3baf61f0441f5c4129 P 78bf5e3ff675477cb9f9c874292fa63f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.672351 29347 tablet_bootstrap.cc:492] T d057ab88d0084efe8d2fe408a42a24b7 P 16f5b5441d8c4a44a9c0a6cca8356822: Bootstrap starting.
I20240501 20:00:34.673020 29348 ts_tablet_manager.cc:1398] T 27c72ce2560f4d3baf61f0441f5c4129 P 78bf5e3ff675477cb9f9c874292fa63f: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20240501 20:00:34.673841 29348 tablet_bootstrap.cc:492] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 78bf5e3ff675477cb9f9c874292fa63f: Bootstrap starting.
I20240501 20:00:34.671037 29355 raft_consensus.cc:504] T eed1ac06c82147e3bb681f4bcd1d8688 P 8feebcdef3ba428ba32e62016f3ed365 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.678525 29347 tablet_bootstrap.cc:654] T d057ab88d0084efe8d2fe408a42a24b7 P 16f5b5441d8c4a44a9c0a6cca8356822: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:34.679539 29231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eed1ac06c82147e3bb681f4bcd1d8688" candidate_uuid: "8feebcdef3ba428ba32e62016f3ed365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78bf5e3ff675477cb9f9c874292fa63f"
I20240501 20:00:34.679980 29307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eed1ac06c82147e3bb681f4bcd1d8688" candidate_uuid: "8feebcdef3ba428ba32e62016f3ed365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16f5b5441d8c4a44a9c0a6cca8356822"
I20240501 20:00:34.680182 29231 raft_consensus.cc:3037] T eed1ac06c82147e3bb681f4bcd1d8688 P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Advancing to term 1
W20240501 20:00:34.681336 29122 leader_election.cc:341] T eed1ac06c82147e3bb681f4bcd1d8688 P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195:37025): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:00:34.681995 29355 leader_election.cc:288] T eed1ac06c82147e3bb681f4bcd1d8688 P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 1 election: Requested vote from peers 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194:39263), 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195:37025)
I20240501 20:00:34.684811 29348 tablet_bootstrap.cc:654] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 78bf5e3ff675477cb9f9c874292fa63f: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:34.686151 29232 raft_consensus.cc:2454] T d057ab88d0084efe8d2fe408a42a24b7 P 78bf5e3ff675477cb9f9c874292fa63f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8feebcdef3ba428ba32e62016f3ed365 in term 1.
I20240501 20:00:34.687502 29231 raft_consensus.cc:2454] T eed1ac06c82147e3bb681f4bcd1d8688 P 78bf5e3ff675477cb9f9c874292fa63f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8feebcdef3ba428ba32e62016f3ed365 in term 1.
I20240501 20:00:34.689002 29391 raft_consensus.cc:482] T 27c72ce2560f4d3baf61f0441f5c4129 P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:34.689452 29391 raft_consensus.cc:504] T 27c72ce2560f4d3baf61f0441f5c4129 P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.692533 29347 tablet_bootstrap.cc:492] T d057ab88d0084efe8d2fe408a42a24b7 P 16f5b5441d8c4a44a9c0a6cca8356822: No bootstrap required, opened a new log
I20240501 20:00:34.692951 29347 ts_tablet_manager.cc:1367] T d057ab88d0084efe8d2fe408a42a24b7 P 16f5b5441d8c4a44a9c0a6cca8356822: Time spent bootstrapping tablet: real 0.021s user 0.012s sys 0.004s
I20240501 20:00:34.692883 29053 catalog_manager.cc:5484] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 8feebcdef3ba428ba32e62016f3ed365 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193). New cstate: current_term: 1 leader_uuid: "8feebcdef3ba428ba32e62016f3ed365" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } health_report { overall_health: UNKNOWN } } }
I20240501 20:00:34.693989 29391 leader_election.cc:288] T 27c72ce2560f4d3baf61f0441f5c4129 P 78bf5e3ff675477cb9f9c874292fa63f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193:43937), 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195:37025)
I20240501 20:00:34.695545 29347 raft_consensus.cc:348] T d057ab88d0084efe8d2fe408a42a24b7 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.696403 29347 raft_consensus.cc:374] T d057ab88d0084efe8d2fe408a42a24b7 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:34.696774 29347 raft_consensus.cc:729] T d057ab88d0084efe8d2fe408a42a24b7 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16f5b5441d8c4a44a9c0a6cca8356822, State: Initialized, Role: FOLLOWER
I20240501 20:00:34.697934 29347 consensus_queue.cc:260] T d057ab88d0084efe8d2fe408a42a24b7 P 16f5b5441d8c4a44a9c0a6cca8356822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.696374 29355 raft_consensus.cc:482] T afbcfea6a7894b5ab7c122d3d4134869 P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:34.701112 29355 raft_consensus.cc:504] T afbcfea6a7894b5ab7c122d3d4134869 P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.701936 29233 raft_consensus.cc:2454] T 419e173751fc4cd18b50341160a84d24 P 78bf5e3ff675477cb9f9c874292fa63f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8feebcdef3ba428ba32e62016f3ed365 in term 1.
I20240501 20:00:34.703122 29355 leader_election.cc:288] T afbcfea6a7894b5ab7c122d3d4134869 P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194:39263), 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195:37025)
I20240501 20:00:34.704298 29307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afbcfea6a7894b5ab7c122d3d4134869" candidate_uuid: "8feebcdef3ba428ba32e62016f3ed365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" is_pre_election: true
I20240501 20:00:34.704331 29124 leader_election.cc:302] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78bf5e3ff675477cb9f9c874292fa63f, 8feebcdef3ba428ba32e62016f3ed365; no voters: 16f5b5441d8c4a44a9c0a6cca8356822
I20240501 20:00:34.707304 29124 leader_election.cc:302] T eed1ac06c82147e3bb681f4bcd1d8688 P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78bf5e3ff675477cb9f9c874292fa63f, 8feebcdef3ba428ba32e62016f3ed365; no voters: 16f5b5441d8c4a44a9c0a6cca8356822
I20240501 20:00:34.708525 29307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c72ce2560f4d3baf61f0441f5c4129" candidate_uuid: "78bf5e3ff675477cb9f9c874292fa63f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" is_pre_election: true
I20240501 20:00:34.709748 29355 raft_consensus.cc:2781] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:00:34.710085 29347 ts_tablet_manager.cc:1398] T d057ab88d0084efe8d2fe408a42a24b7 P 16f5b5441d8c4a44a9c0a6cca8356822: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20240501 20:00:34.709797 29158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c72ce2560f4d3baf61f0441f5c4129" candidate_uuid: "78bf5e3ff675477cb9f9c874292fa63f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8feebcdef3ba428ba32e62016f3ed365" is_pre_election: true
I20240501 20:00:34.710431 29398 raft_consensus.cc:2781] T eed1ac06c82147e3bb681f4bcd1d8688 P 8feebcdef3ba428ba32e62016f3ed365 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:00:34.710884 29398 raft_consensus.cc:686] T eed1ac06c82147e3bb681f4bcd1d8688 P 8feebcdef3ba428ba32e62016f3ed365 [term 1 LEADER]: Becoming Leader. State: Replica: 8feebcdef3ba428ba32e62016f3ed365, State: Running, Role: LEADER
I20240501 20:00:34.710893 29158 raft_consensus.cc:2379] T 27c72ce2560f4d3baf61f0441f5c4129 P 8feebcdef3ba428ba32e62016f3ed365 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 78bf5e3ff675477cb9f9c874292fa63f in current term 1: Already voted for candidate 16f5b5441d8c4a44a9c0a6cca8356822 in this term.
I20240501 20:00:34.711598 29124 leader_election.cc:302] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 78bf5e3ff675477cb9f9c874292fa63f, 8feebcdef3ba428ba32e62016f3ed365; no voters: 16f5b5441d8c4a44a9c0a6cca8356822
I20240501 20:00:34.710958 29347 tablet_bootstrap.cc:492] T eed1ac06c82147e3bb681f4bcd1d8688 P 16f5b5441d8c4a44a9c0a6cca8356822: Bootstrap starting.
I20240501 20:00:34.715340 29397 raft_consensus.cc:2781] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:00:34.712328 29355 raft_consensus.cc:686] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365 [term 1 LEADER]: Becoming Leader. State: Replica: 8feebcdef3ba428ba32e62016f3ed365, State: Running, Role: LEADER
I20240501 20:00:34.720360 29397 raft_consensus.cc:686] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 [term 1 LEADER]: Becoming Leader. State: Replica: 8feebcdef3ba428ba32e62016f3ed365, State: Running, Role: LEADER
I20240501 20:00:34.730861 29397 consensus_queue.cc:237] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.711777 29398 consensus_queue.cc:237] T eed1ac06c82147e3bb681f4bcd1d8688 P 8feebcdef3ba428ba32e62016f3ed365 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.736263 29372 raft_consensus.cc:482] T 419e173751fc4cd18b50341160a84d24 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:34.736744 29372 raft_consensus.cc:504] T 419e173751fc4cd18b50341160a84d24 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.737933 29355 consensus_queue.cc:237] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.742168 29158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "419e173751fc4cd18b50341160a84d24" candidate_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8feebcdef3ba428ba32e62016f3ed365" is_pre_election: true
I20240501 20:00:34.742221 29197 leader_election.cc:302] T 27c72ce2560f4d3baf61f0441f5c4129 P 78bf5e3ff675477cb9f9c874292fa63f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 78bf5e3ff675477cb9f9c874292fa63f; no voters: 16f5b5441d8c4a44a9c0a6cca8356822, 8feebcdef3ba428ba32e62016f3ed365
I20240501 20:00:34.743314 29391 raft_consensus.cc:3037] T 27c72ce2560f4d3baf61f0441f5c4129 P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:34.712328 29352 raft_consensus.cc:504] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.750378 29352 leader_election.cc:288] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f [CANDIDATE]: Term 1 election: Requested vote from peers 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193:43937), 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195:37025)
I20240501 20:00:34.751224 29158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bd59b3e54144bd2986538a355cadd9b" candidate_uuid: "78bf5e3ff675477cb9f9c874292fa63f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8feebcdef3ba428ba32e62016f3ed365"
I20240501 20:00:34.751997 29158 raft_consensus.cc:3037] T 5bd59b3e54144bd2986538a355cadd9b P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:34.742168 29233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afbcfea6a7894b5ab7c122d3d4134869" candidate_uuid: "8feebcdef3ba428ba32e62016f3ed365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78bf5e3ff675477cb9f9c874292fa63f" is_pre_election: true
I20240501 20:00:34.757565 29233 raft_consensus.cc:2379] T afbcfea6a7894b5ab7c122d3d4134869 P 78bf5e3ff675477cb9f9c874292fa63f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8feebcdef3ba428ba32e62016f3ed365 in current term 1: Already voted for candidate 16f5b5441d8c4a44a9c0a6cca8356822 in this term.
I20240501 20:00:34.759611 29372 leader_election.cc:288] T 419e173751fc4cd18b50341160a84d24 P 16f5b5441d8c4a44a9c0a6cca8356822 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193:43937), 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194:39263)
I20240501 20:00:34.766237 29347 tablet_bootstrap.cc:654] T eed1ac06c82147e3bb681f4bcd1d8688 P 16f5b5441d8c4a44a9c0a6cca8356822: Neither blocks nor log segments found. Creating new log.
I20240501 20:00:34.768464 29233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "419e173751fc4cd18b50341160a84d24" candidate_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78bf5e3ff675477cb9f9c874292fa63f" is_pre_election: true
I20240501 20:00:34.772542 29233 raft_consensus.cc:2379] T 419e173751fc4cd18b50341160a84d24 P 78bf5e3ff675477cb9f9c874292fa63f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 16f5b5441d8c4a44a9c0a6cca8356822 in current term 1: Already voted for candidate 8feebcdef3ba428ba32e62016f3ed365 in this term.
I20240501 20:00:34.776221 29348 tablet_bootstrap.cc:492] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 78bf5e3ff675477cb9f9c874292fa63f: No bootstrap required, opened a new log
I20240501 20:00:34.776684 29348 ts_tablet_manager.cc:1367] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 78bf5e3ff675477cb9f9c874292fa63f: Time spent bootstrapping tablet: real 0.103s user 0.026s sys 0.035s
W20240501 20:00:34.776578 29124 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:39263, user_credentials={real_user=slave}} blocked reactor thread for 65071.8us
I20240501 20:00:34.776975 29273 leader_election.cc:302] T 419e173751fc4cd18b50341160a84d24 P 16f5b5441d8c4a44a9c0a6cca8356822 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 16f5b5441d8c4a44a9c0a6cca8356822; no voters: 78bf5e3ff675477cb9f9c874292fa63f, 8feebcdef3ba428ba32e62016f3ed365
I20240501 20:00:34.777420 29382 raft_consensus.cc:482] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:34.777633 29124 leader_election.cc:302] T afbcfea6a7894b5ab7c122d3d4134869 P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8feebcdef3ba428ba32e62016f3ed365; no voters: 16f5b5441d8c4a44a9c0a6cca8356822, 78bf5e3ff675477cb9f9c874292fa63f
I20240501 20:00:34.777863 29382 raft_consensus.cc:504] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } } peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } }
I20240501 20:00:34.778623 29372 raft_consensus.cc:3037] T 419e173751fc4cd18b50341160a84d24 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:34.781888 29391 raft_consensus.cc:2726] T 27c72ce2560f4d3baf61f0441f5c4129 P 78bf5e3ff675477cb9f9c874292fa63f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:00:34.782660 29348 raft_consensus.cc:348] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.782773 29306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d9b61e9c98e4d19b9272fe70fab0c3d" candidate_uuid: "8feebcdef3ba428ba32e62016f3ed365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" is_pre_election: true
I20240501 20:00:34.783386 29348 raft_consensus.cc:374] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:34.783690 29348 raft_consensus.cc:729] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78bf5e3ff675477cb9f9c874292fa63f, State: Initialized, Role: FOLLOWER
I20240501 20:00:34.783907 29306 raft_consensus.cc:2379] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 16f5b5441d8c4a44a9c0a6cca8356822 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8feebcdef3ba428ba32e62016f3ed365 in current term 1: Already voted for candidate 78bf5e3ff675477cb9f9c874292fa63f in this term.
I20240501 20:00:34.784226 29307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bd59b3e54144bd2986538a355cadd9b" candidate_uuid: "78bf5e3ff675477cb9f9c874292fa63f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16f5b5441d8c4a44a9c0a6cca8356822"
I20240501 20:00:34.784801 29307 raft_consensus.cc:3037] T 5bd59b3e54144bd2986538a355cadd9b P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:34.784431 29348 consensus_queue.cc:260] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 78bf5e3ff675477cb9f9c874292fa63f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.787334 29382 leader_election.cc:288] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195:37025), 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194:39263)
I20240501 20:00:34.787525 29406 raft_consensus.cc:3037] T afbcfea6a7894b5ab7c122d3d4134869 P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:34.791070 29349 tablet_bootstrap.cc:492] T 3d51e8d55df849e5ac6e5acdc42ac63c P 8feebcdef3ba428ba32e62016f3ed365: No bootstrap required, opened a new log
I20240501 20:00:34.791775 29349 ts_tablet_manager.cc:1367] T 3d51e8d55df849e5ac6e5acdc42ac63c P 8feebcdef3ba428ba32e62016f3ed365: Time spent bootstrapping tablet: real 0.174s user 0.025s sys 0.035s
I20240501 20:00:34.793205 29158 raft_consensus.cc:2454] T 5bd59b3e54144bd2986538a355cadd9b P 8feebcdef3ba428ba32e62016f3ed365 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78bf5e3ff675477cb9f9c874292fa63f in term 1.
I20240501 20:00:34.794900 29349 raft_consensus.cc:348] T 3d51e8d55df849e5ac6e5acdc42ac63c P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.755676 29053 catalog_manager.cc:5484] T 27c72ce2560f4d3baf61f0441f5c4129 P 16f5b5441d8c4a44a9c0a6cca8356822 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195). New cstate: current_term: 1 leader_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } health_report { overall_health: HEALTHY } } }
I20240501 20:00:34.794183 29233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d9b61e9c98e4d19b9272fe70fab0c3d" candidate_uuid: "8feebcdef3ba428ba32e62016f3ed365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78bf5e3ff675477cb9f9c874292fa63f" is_pre_election: true
I20240501 20:00:34.795935 29199 leader_election.cc:302] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78bf5e3ff675477cb9f9c874292fa63f, 8feebcdef3ba428ba32e62016f3ed365; no voters:
I20240501 20:00:34.796586 29349 raft_consensus.cc:374] T 3d51e8d55df849e5ac6e5acdc42ac63c P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:34.796970 29349 raft_consensus.cc:729] T 3d51e8d55df849e5ac6e5acdc42ac63c P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8feebcdef3ba428ba32e62016f3ed365, State: Initialized, Role: FOLLOWER
I20240501 20:00:34.800508 29391 raft_consensus.cc:2781] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:00:34.796790 29053 catalog_manager.cc:5484] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195). New cstate: current_term: 1 leader_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } health_report { overall_health: HEALTHY } } }
I20240501 20:00:34.806782 29391 raft_consensus.cc:686] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f [term 1 LEADER]: Becoming Leader. State: Replica: 78bf5e3ff675477cb9f9c874292fa63f, State: Running, Role: LEADER
I20240501 20:00:34.797297 29124 leader_election.cc:302] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8feebcdef3ba428ba32e62016f3ed365; no voters: 16f5b5441d8c4a44a9c0a6cca8356822, 78bf5e3ff675477cb9f9c874292fa63f
I20240501 20:00:34.831452 29348 ts_tablet_manager.cc:1398] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 78bf5e3ff675477cb9f9c874292fa63f: Time spent starting tablet: real 0.054s user 0.007s sys 0.000s
I20240501 20:00:34.837227 29053 catalog_manager.cc:5484] T afbcfea6a7894b5ab7c122d3d4134869 P 16f5b5441d8c4a44a9c0a6cca8356822 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195). New cstate: current_term: 1 leader_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } health_report { overall_health: HEALTHY } } }
I20240501 20:00:34.850149 29355 raft_consensus.cc:3037] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:34.847174 29056 catalog_manager.cc:5484] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193). New cstate: current_term: 1 leader_uuid: "8feebcdef3ba428ba32e62016f3ed365" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } health_report { overall_health: UNKNOWN } } }
I20240501 20:00:34.812031 29391 consensus_queue.cc:237] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.864224 29349 consensus_queue.cc:260] T 3d51e8d55df849e5ac6e5acdc42ac63c P 8feebcdef3ba428ba32e62016f3ed365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
W20240501 20:00:34.864801 29124 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:39263, user_credentials={real_user=slave}} blocked reactor thread for 67593.8us
I20240501 20:00:34.866417 29354 consensus_queue.cc:1035] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:00:34.873034 29355 raft_consensus.cc:2726] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 8feebcdef3ba428ba32e62016f3ed365 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:00:34.877238 29372 raft_consensus.cc:2726] T 419e173751fc4cd18b50341160a84d24 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:00:34.869829 29406 raft_consensus.cc:2726] T afbcfea6a7894b5ab7c122d3d4134869 P 8feebcdef3ba428ba32e62016f3ed365 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:00:34.879724 29347 tablet_bootstrap.cc:492] T eed1ac06c82147e3bb681f4bcd1d8688 P 16f5b5441d8c4a44a9c0a6cca8356822: No bootstrap required, opened a new log
I20240501 20:00:34.882155 29347 ts_tablet_manager.cc:1367] T eed1ac06c82147e3bb681f4bcd1d8688 P 16f5b5441d8c4a44a9c0a6cca8356822: Time spent bootstrapping tablet: real 0.171s user 0.026s sys 0.023s
I20240501 20:00:34.881189 29349 ts_tablet_manager.cc:1398] T 3d51e8d55df849e5ac6e5acdc42ac63c P 8feebcdef3ba428ba32e62016f3ed365: Time spent starting tablet: real 0.089s user 0.004s sys 0.003s
I20240501 20:00:34.883968 29347 raft_consensus.cc:348] T eed1ac06c82147e3bb681f4bcd1d8688 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.884734 29347 raft_consensus.cc:374] T eed1ac06c82147e3bb681f4bcd1d8688 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:00:34.885058 29347 raft_consensus.cc:729] T eed1ac06c82147e3bb681f4bcd1d8688 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16f5b5441d8c4a44a9c0a6cca8356822, State: Initialized, Role: FOLLOWER
I20240501 20:00:34.885588 29347 consensus_queue.cc:260] T eed1ac06c82147e3bb681f4bcd1d8688 P 16f5b5441d8c4a44a9c0a6cca8356822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:34.887109 29347 ts_tablet_manager.cc:1398] T eed1ac06c82147e3bb681f4bcd1d8688 P 16f5b5441d8c4a44a9c0a6cca8356822: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:00:34.880157 29307 raft_consensus.cc:2454] T 5bd59b3e54144bd2986538a355cadd9b P 16f5b5441d8c4a44a9c0a6cca8356822 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78bf5e3ff675477cb9f9c874292fa63f in term 1.
I20240501 20:00:34.894836 29056 catalog_manager.cc:5484] T eed1ac06c82147e3bb681f4bcd1d8688 P 8feebcdef3ba428ba32e62016f3ed365 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193). New cstate: current_term: 1 leader_uuid: "8feebcdef3ba428ba32e62016f3ed365" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } health_report { overall_health: UNKNOWN } } }
I20240501 20:00:34.896524 29056 catalog_manager.cc:5484] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193). New cstate: current_term: 1 leader_uuid: "8feebcdef3ba428ba32e62016f3ed365" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } health_report { overall_health: UNKNOWN } } }
I20240501 20:00:34.903903 29158 raft_consensus.cc:3037] T 3d51e8d55df849e5ac6e5acdc42ac63c P 8feebcdef3ba428ba32e62016f3ed365 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:34.907739 29054 catalog_manager.cc:5484] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f reported cstate change: term changed from 0 to 1, leader changed from <none> to 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194). New cstate: current_term: 1 leader_uuid: "78bf5e3ff675477cb9f9c874292fa63f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } health_report { overall_health: UNKNOWN } } }
I20240501 20:00:34.911592 29372 consensus_queue.cc:1035] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:00:34.923609 29372 consensus_queue.cc:1035] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:00:34.934695 29354 consensus_queue.cc:1035] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:00:35.043444 29352 consensus_queue.cc:1035] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 78bf5e3ff675477cb9f9c874292fa63f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:00:35.053009 29352 consensus_queue.cc:1035] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 78bf5e3ff675477cb9f9c874292fa63f [LEADER]: Connected to new peer: Peer: permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:00:35.080654 29233 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0d9b61e9c98e4d19b9272fe70fab0c3d"
dest_uuid: "78bf5e3ff675477cb9f9c874292fa63f"
mode: GRACEFUL
new_leader_uuid: "8feebcdef3ba428ba32e62016f3ed365"
from {username='slave'} at 127.0.0.1:41300
I20240501 20:00:35.081322 29233 raft_consensus.cc:595] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 78bf5e3ff675477cb9f9c874292fa63f [term 1 LEADER]: Received request to transfer leadership to TS 8feebcdef3ba428ba32e62016f3ed365
I20240501 20:00:35.081465 29418 consensus_queue.cc:1035] T afbcfea6a7894b5ab7c122d3d4134869 P 16f5b5441d8c4a44a9c0a6cca8356822 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:00:35.081699 29373 raft_consensus.cc:482] T d057ab88d0084efe8d2fe408a42a24b7 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:35.082832 29373 raft_consensus.cc:504] T d057ab88d0084efe8d2fe408a42a24b7 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:35.084636 29373 leader_election.cc:288] T d057ab88d0084efe8d2fe408a42a24b7 P 16f5b5441d8c4a44a9c0a6cca8356822 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193:43937), 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194:39263)
I20240501 20:00:35.085664 29233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d057ab88d0084efe8d2fe408a42a24b7" candidate_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78bf5e3ff675477cb9f9c874292fa63f" is_pre_election: true
I20240501 20:00:35.085791 29158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d057ab88d0084efe8d2fe408a42a24b7" candidate_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8feebcdef3ba428ba32e62016f3ed365" is_pre_election: true
I20240501 20:00:35.086468 29233 raft_consensus.cc:2379] T d057ab88d0084efe8d2fe408a42a24b7 P 78bf5e3ff675477cb9f9c874292fa63f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 16f5b5441d8c4a44a9c0a6cca8356822 in current term 1: Already voted for candidate 8feebcdef3ba428ba32e62016f3ed365 in this term.
I20240501 20:00:35.087494 29273 leader_election.cc:302] T d057ab88d0084efe8d2fe408a42a24b7 P 16f5b5441d8c4a44a9c0a6cca8356822 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 16f5b5441d8c4a44a9c0a6cca8356822; no voters: 78bf5e3ff675477cb9f9c874292fa63f, 8feebcdef3ba428ba32e62016f3ed365
I20240501 20:00:35.089128 29418 raft_consensus.cc:3037] T d057ab88d0084efe8d2fe408a42a24b7 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:35.095487 29418 raft_consensus.cc:2726] T d057ab88d0084efe8d2fe408a42a24b7 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:00:35.098564 29373 consensus_queue.cc:1035] T afbcfea6a7894b5ab7c122d3d4134869 P 16f5b5441d8c4a44a9c0a6cca8356822 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:00:35.099432 29307 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3d51e8d55df849e5ac6e5acdc42ac63c"
dest_uuid: "16f5b5441d8c4a44a9c0a6cca8356822"
mode: GRACEFUL
new_leader_uuid: "8feebcdef3ba428ba32e62016f3ed365"
from {username='slave'} at 127.0.0.1:56748
I20240501 20:00:35.100075 29307 raft_consensus.cc:595] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 [term 1 LEADER]: Received request to transfer leadership to TS 8feebcdef3ba428ba32e62016f3ed365
I20240501 20:00:35.106662 29307 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f4895041b0924276aa587adbb8dc9e98"
dest_uuid: "16f5b5441d8c4a44a9c0a6cca8356822"
mode: GRACEFUL
new_leader_uuid: "8feebcdef3ba428ba32e62016f3ed365"
from {username='slave'} at 127.0.0.1:56748
I20240501 20:00:35.107254 29307 raft_consensus.cc:595] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 1 LEADER]: Received request to transfer leadership to TS 8feebcdef3ba428ba32e62016f3ed365
I20240501 20:00:35.112939 29307 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "afbcfea6a7894b5ab7c122d3d4134869"
dest_uuid: "16f5b5441d8c4a44a9c0a6cca8356822"
mode: GRACEFUL
new_leader_uuid: "8feebcdef3ba428ba32e62016f3ed365"
from {username='slave'} at 127.0.0.1:56748
I20240501 20:00:35.113523 29307 raft_consensus.cc:595] T afbcfea6a7894b5ab7c122d3d4134869 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 1 LEADER]: Received request to transfer leadership to TS 8feebcdef3ba428ba32e62016f3ed365
I20240501 20:00:35.118454 29233 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5bd59b3e54144bd2986538a355cadd9b"
dest_uuid: "78bf5e3ff675477cb9f9c874292fa63f"
mode: GRACEFUL
new_leader_uuid: "8feebcdef3ba428ba32e62016f3ed365"
from {username='slave'} at 127.0.0.1:41300
I20240501 20:00:35.118988 29233 raft_consensus.cc:595] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f [term 1 LEADER]: Received request to transfer leadership to TS 8feebcdef3ba428ba32e62016f3ed365
I20240501 20:00:35.122920 29307 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "27c72ce2560f4d3baf61f0441f5c4129"
dest_uuid: "16f5b5441d8c4a44a9c0a6cca8356822"
mode: GRACEFUL
new_leader_uuid: "78bf5e3ff675477cb9f9c874292fa63f"
from {username='slave'} at 127.0.0.1:56748
I20240501 20:00:35.123350 29307 raft_consensus.cc:595] T 27c72ce2560f4d3baf61f0441f5c4129 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 1 LEADER]: Received request to transfer leadership to TS 78bf5e3ff675477cb9f9c874292fa63f
I20240501 20:00:35.135691 29157 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "419e173751fc4cd18b50341160a84d24"
dest_uuid: "8feebcdef3ba428ba32e62016f3ed365"
mode: GRACEFUL
new_leader_uuid: "78bf5e3ff675477cb9f9c874292fa63f"
from {username='slave'} at 127.0.0.1:48850
I20240501 20:00:35.136257 29157 raft_consensus.cc:595] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 [term 1 LEADER]: Received request to transfer leadership to TS 78bf5e3ff675477cb9f9c874292fa63f
I20240501 20:00:35.140650 29157 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d057ab88d0084efe8d2fe408a42a24b7"
dest_uuid: "8feebcdef3ba428ba32e62016f3ed365"
mode: GRACEFUL
new_leader_uuid: "78bf5e3ff675477cb9f9c874292fa63f"
from {username='slave'} at 127.0.0.1:48850
I20240501 20:00:35.141116 29157 raft_consensus.cc:595] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365 [term 1 LEADER]: Received request to transfer leadership to TS 78bf5e3ff675477cb9f9c874292fa63f
I20240501 20:00:35.145160 29157 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eed1ac06c82147e3bb681f4bcd1d8688"
dest_uuid: "8feebcdef3ba428ba32e62016f3ed365"
mode: GRACEFUL
new_leader_uuid: "78bf5e3ff675477cb9f9c874292fa63f"
from {username='slave'} at 127.0.0.1:48850
I20240501 20:00:35.145596 29157 raft_consensus.cc:595] T eed1ac06c82147e3bb681f4bcd1d8688 P 8feebcdef3ba428ba32e62016f3ed365 [term 1 LEADER]: Received request to transfer leadership to TS 78bf5e3ff675477cb9f9c874292fa63f
I20240501 20:00:35.149787 29157 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6c7b0c234f7d4cc0a0848e47e24c5d8b"
dest_uuid: "8feebcdef3ba428ba32e62016f3ed365"
mode: GRACEFUL
new_leader_uuid: "16f5b5441d8c4a44a9c0a6cca8356822"
from {username='slave'} at 127.0.0.1:48850
I20240501 20:00:35.150220 29157 raft_consensus.cc:595] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 8feebcdef3ba428ba32e62016f3ed365 [term 1 LEADER]: Received request to transfer leadership to TS 16f5b5441d8c4a44a9c0a6cca8356822
I20240501 20:00:35.172917 29233 raft_consensus.cc:3037] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 78bf5e3ff675477cb9f9c874292fa63f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:35.178193 29406 consensus_queue.cc:1035] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 8feebcdef3ba428ba32e62016f3ed365 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:00:35.187424 29355 consensus_queue.cc:1035] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 8feebcdef3ba428ba32e62016f3ed365 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:00:35.194341 29397 raft_consensus.cc:979] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 8feebcdef3ba428ba32e62016f3ed365: : Instructing follower 16f5b5441d8c4a44a9c0a6cca8356822 to start an election
I20240501 20:00:35.194797 29397 raft_consensus.cc:1067] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 8feebcdef3ba428ba32e62016f3ed365 [term 1 LEADER]: Signalling peer 16f5b5441d8c4a44a9c0a6cca8356822 to start an election
I20240501 20:00:35.196235 29307 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6c7b0c234f7d4cc0a0848e47e24c5d8b"
dest_uuid: "16f5b5441d8c4a44a9c0a6cca8356822"
from {username='slave'} at 127.0.0.1:39788
I20240501 20:00:35.196671 29307 raft_consensus.cc:482] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 16f5b5441d8c4a44a9c0a6cca8356822 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:00:35.196966 29307 raft_consensus.cc:3037] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 16f5b5441d8c4a44a9c0a6cca8356822 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:35.201251 29307 raft_consensus.cc:504] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 16f5b5441d8c4a44a9c0a6cca8356822 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:35.202654 29307 leader_election.cc:288] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 16f5b5441d8c4a44a9c0a6cca8356822 [CANDIDATE]: Term 2 election: Requested vote from peers 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193:43937), 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194:39263)
I20240501 20:00:35.203462 29157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c7b0c234f7d4cc0a0848e47e24c5d8b" candidate_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8feebcdef3ba428ba32e62016f3ed365"
I20240501 20:00:35.203816 29233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c7b0c234f7d4cc0a0848e47e24c5d8b" candidate_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "78bf5e3ff675477cb9f9c874292fa63f"
I20240501 20:00:35.204037 29157 raft_consensus.cc:3032] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 8feebcdef3ba428ba32e62016f3ed365 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:00:35.204386 29157 raft_consensus.cc:729] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 8feebcdef3ba428ba32e62016f3ed365 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8feebcdef3ba428ba32e62016f3ed365, State: Running, Role: LEADER
I20240501 20:00:35.204430 29233 raft_consensus.cc:3037] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 78bf5e3ff675477cb9f9c874292fa63f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:35.205225 29157 consensus_queue.cc:260] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 8feebcdef3ba428ba32e62016f3ed365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:35.206439 29157 raft_consensus.cc:3037] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 8feebcdef3ba428ba32e62016f3ed365 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:35.210999 29233 raft_consensus.cc:2454] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 78bf5e3ff675477cb9f9c874292fa63f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16f5b5441d8c4a44a9c0a6cca8356822 in term 2.
I20240501 20:00:35.212038 29273 leader_election.cc:302] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 16f5b5441d8c4a44a9c0a6cca8356822 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16f5b5441d8c4a44a9c0a6cca8356822, 78bf5e3ff675477cb9f9c874292fa63f; no voters:
I20240501 20:00:35.212843 29418 raft_consensus.cc:2781] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 16f5b5441d8c4a44a9c0a6cca8356822 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:00:35.213045 29157 raft_consensus.cc:2454] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 8feebcdef3ba428ba32e62016f3ed365 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16f5b5441d8c4a44a9c0a6cca8356822 in term 2.
I20240501 20:00:35.213289 29418 raft_consensus.cc:686] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 16f5b5441d8c4a44a9c0a6cca8356822 [term 2 LEADER]: Becoming Leader. State: Replica: 16f5b5441d8c4a44a9c0a6cca8356822, State: Running, Role: LEADER
I20240501 20:00:35.214156 29418 consensus_queue.cc:237] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 16f5b5441d8c4a44a9c0a6cca8356822 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:35.217770 29418 consensus_queue.cc:1035] T 27c72ce2560f4d3baf61f0441f5c4129 P 16f5b5441d8c4a44a9c0a6cca8356822 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:00:35.221118 29054 catalog_manager.cc:5484] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 16f5b5441d8c4a44a9c0a6cca8356822 reported cstate change: term changed from 1 to 2, leader changed from 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193) to 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195). New cstate: current_term: 2 leader_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } health_report { overall_health: HEALTHY } } }
I20240501 20:00:35.227829 29418 consensus_queue.cc:1035] T 27c72ce2560f4d3baf61f0441f5c4129 P 16f5b5441d8c4a44a9c0a6cca8356822 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:00:35.236569 29372 raft_consensus.cc:979] T 27c72ce2560f4d3baf61f0441f5c4129 P 16f5b5441d8c4a44a9c0a6cca8356822: : Instructing follower 78bf5e3ff675477cb9f9c874292fa63f to start an election
I20240501 20:00:35.236975 29418 raft_consensus.cc:1067] T 27c72ce2560f4d3baf61f0441f5c4129 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 1 LEADER]: Signalling peer 78bf5e3ff675477cb9f9c874292fa63f to start an election
I20240501 20:00:35.238355 29233 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "27c72ce2560f4d3baf61f0441f5c4129"
dest_uuid: "78bf5e3ff675477cb9f9c874292fa63f"
from {username='slave'} at 127.0.0.1:42824
I20240501 20:00:35.238952 29233 raft_consensus.cc:482] T 27c72ce2560f4d3baf61f0441f5c4129 P 78bf5e3ff675477cb9f9c874292fa63f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:00:35.239419 29233 raft_consensus.cc:3037] T 27c72ce2560f4d3baf61f0441f5c4129 P 78bf5e3ff675477cb9f9c874292fa63f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:35.241329 29397 consensus_queue.cc:1035] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:00:35.245836 29233 raft_consensus.cc:504] T 27c72ce2560f4d3baf61f0441f5c4129 P 78bf5e3ff675477cb9f9c874292fa63f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:35.247113 29397 consensus_queue.cc:1035] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:00:35.248538 29157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c72ce2560f4d3baf61f0441f5c4129" candidate_uuid: "78bf5e3ff675477cb9f9c874292fa63f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8feebcdef3ba428ba32e62016f3ed365"
I20240501 20:00:35.248682 29233 leader_election.cc:288] T 27c72ce2560f4d3baf61f0441f5c4129 P 78bf5e3ff675477cb9f9c874292fa63f [CANDIDATE]: Term 2 election: Requested vote from peers 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193:43937), 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195:37025)
I20240501 20:00:35.249130 29157 raft_consensus.cc:3037] T 27c72ce2560f4d3baf61f0441f5c4129 P 8feebcdef3ba428ba32e62016f3ed365 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:35.250579 29307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c72ce2560f4d3baf61f0441f5c4129" candidate_uuid: "78bf5e3ff675477cb9f9c874292fa63f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "16f5b5441d8c4a44a9c0a6cca8356822"
I20240501 20:00:35.251133 29307 raft_consensus.cc:3032] T 27c72ce2560f4d3baf61f0441f5c4129 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:00:35.251446 29307 raft_consensus.cc:729] T 27c72ce2560f4d3baf61f0441f5c4129 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 16f5b5441d8c4a44a9c0a6cca8356822, State: Running, Role: LEADER
I20240501 20:00:35.252226 29307 consensus_queue.cc:260] T 27c72ce2560f4d3baf61f0441f5c4129 P 16f5b5441d8c4a44a9c0a6cca8356822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:35.253446 29307 raft_consensus.cc:3037] T 27c72ce2560f4d3baf61f0441f5c4129 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:35.248663 29355 raft_consensus.cc:979] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365: : Instructing follower 78bf5e3ff675477cb9f9c874292fa63f to start an election
I20240501 20:00:35.254148 29355 raft_consensus.cc:1067] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365 [term 1 LEADER]: Signalling peer 78bf5e3ff675477cb9f9c874292fa63f to start an election
I20240501 20:00:35.257454 29157 raft_consensus.cc:2454] T 27c72ce2560f4d3baf61f0441f5c4129 P 8feebcdef3ba428ba32e62016f3ed365 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78bf5e3ff675477cb9f9c874292fa63f in term 2.
I20240501 20:00:35.257836 29231 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d057ab88d0084efe8d2fe408a42a24b7"
dest_uuid: "78bf5e3ff675477cb9f9c874292fa63f"
from {username='slave'} at 127.0.0.1:42828
I20240501 20:00:35.258567 29231 raft_consensus.cc:482] T d057ab88d0084efe8d2fe408a42a24b7 P 78bf5e3ff675477cb9f9c874292fa63f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:00:35.259012 29231 raft_consensus.cc:3037] T d057ab88d0084efe8d2fe408a42a24b7 P 78bf5e3ff675477cb9f9c874292fa63f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:35.259251 29382 consensus_queue.cc:1035] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:00:35.266112 29199 leader_election.cc:302] T 27c72ce2560f4d3baf61f0441f5c4129 P 78bf5e3ff675477cb9f9c874292fa63f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78bf5e3ff675477cb9f9c874292fa63f, 8feebcdef3ba428ba32e62016f3ed365; no voters:
I20240501 20:00:35.266434 29307 raft_consensus.cc:2454] T 27c72ce2560f4d3baf61f0441f5c4129 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78bf5e3ff675477cb9f9c874292fa63f in term 2.
I20240501 20:00:35.267128 29352 raft_consensus.cc:2781] T 27c72ce2560f4d3baf61f0441f5c4129 P 78bf5e3ff675477cb9f9c874292fa63f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:00:35.267638 29352 raft_consensus.cc:686] T 27c72ce2560f4d3baf61f0441f5c4129 P 78bf5e3ff675477cb9f9c874292fa63f [term 2 LEADER]: Becoming Leader. State: Replica: 78bf5e3ff675477cb9f9c874292fa63f, State: Running, Role: LEADER
I20240501 20:00:35.268339 29352 consensus_queue.cc:237] T 27c72ce2560f4d3baf61f0441f5c4129 P 78bf5e3ff675477cb9f9c874292fa63f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:35.268808 29418 raft_consensus.cc:482] T eed1ac06c82147e3bb681f4bcd1d8688 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:35.269253 29418 raft_consensus.cc:504] T eed1ac06c82147e3bb681f4bcd1d8688 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:35.270686 29231 raft_consensus.cc:504] T d057ab88d0084efe8d2fe408a42a24b7 P 78bf5e3ff675477cb9f9c874292fa63f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:35.271164 29418 leader_election.cc:288] T eed1ac06c82147e3bb681f4bcd1d8688 P 16f5b5441d8c4a44a9c0a6cca8356822 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193:43937), 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194:39263)
I20240501 20:00:35.272827 29157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eed1ac06c82147e3bb681f4bcd1d8688" candidate_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8feebcdef3ba428ba32e62016f3ed365" is_pre_election: true
I20240501 20:00:35.273910 29382 consensus_queue.cc:1035] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:00:35.275921 29355 raft_consensus.cc:979] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365: : Instructing follower 78bf5e3ff675477cb9f9c874292fa63f to start an election
I20240501 20:00:35.276304 29157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d057ab88d0084efe8d2fe408a42a24b7" candidate_uuid: "78bf5e3ff675477cb9f9c874292fa63f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8feebcdef3ba428ba32e62016f3ed365"
I20240501 20:00:35.276933 29157 raft_consensus.cc:3032] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:00:35.277254 29157 raft_consensus.cc:729] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8feebcdef3ba428ba32e62016f3ed365, State: Running, Role: LEADER
I20240501 20:00:35.277576 29233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eed1ac06c82147e3bb681f4bcd1d8688" candidate_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78bf5e3ff675477cb9f9c874292fa63f" is_pre_election: true
I20240501 20:00:35.278481 29233 raft_consensus.cc:2379] T eed1ac06c82147e3bb681f4bcd1d8688 P 78bf5e3ff675477cb9f9c874292fa63f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 16f5b5441d8c4a44a9c0a6cca8356822 in current term 1: Already voted for candidate 8feebcdef3ba428ba32e62016f3ed365 in this term.
I20240501 20:00:35.278177 29157 consensus_queue.cc:260] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:35.279460 29157 raft_consensus.cc:3037] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:35.279917 29273 leader_election.cc:302] T eed1ac06c82147e3bb681f4bcd1d8688 P 16f5b5441d8c4a44a9c0a6cca8356822 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 16f5b5441d8c4a44a9c0a6cca8356822; no voters: 78bf5e3ff675477cb9f9c874292fa63f, 8feebcdef3ba428ba32e62016f3ed365
I20240501 20:00:35.280252 29307 raft_consensus.cc:3037] T eed1ac06c82147e3bb681f4bcd1d8688 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:00:35.276413 29355 raft_consensus.cc:1067] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 [term 1 LEADER]: Signalling peer 78bf5e3ff675477cb9f9c874292fa63f to start an election
I20240501 20:00:35.276988 29231 leader_election.cc:288] T d057ab88d0084efe8d2fe408a42a24b7 P 78bf5e3ff675477cb9f9c874292fa63f [CANDIDATE]: Term 2 election: Requested vote from peers 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193:43937), 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195:37025)
I20240501 20:00:35.288596 29306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d057ab88d0084efe8d2fe408a42a24b7" candidate_uuid: "78bf5e3ff675477cb9f9c874292fa63f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "16f5b5441d8c4a44a9c0a6cca8356822"
I20240501 20:00:35.289268 29306 raft_consensus.cc:3037] T d057ab88d0084efe8d2fe408a42a24b7 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:35.289655 29418 raft_consensus.cc:2726] T eed1ac06c82147e3bb681f4bcd1d8688 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:00:35.292137 29355 consensus_queue.cc:1035] T eed1ac06c82147e3bb681f4bcd1d8688 P 8feebcdef3ba428ba32e62016f3ed365 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:00:35.287138 29157 raft_consensus.cc:2454] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78bf5e3ff675477cb9f9c874292fa63f in term 2.
I20240501 20:00:35.298671 29199 leader_election.cc:302] T d057ab88d0084efe8d2fe408a42a24b7 P 78bf5e3ff675477cb9f9c874292fa63f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78bf5e3ff675477cb9f9c874292fa63f, 8feebcdef3ba428ba32e62016f3ed365; no voters:
I20240501 20:00:35.299685 29352 raft_consensus.cc:2781] T d057ab88d0084efe8d2fe408a42a24b7 P 78bf5e3ff675477cb9f9c874292fa63f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:00:35.301175 29352 raft_consensus.cc:686] T d057ab88d0084efe8d2fe408a42a24b7 P 78bf5e3ff675477cb9f9c874292fa63f [term 2 LEADER]: Becoming Leader. State: Replica: 78bf5e3ff675477cb9f9c874292fa63f, State: Running, Role: LEADER
I20240501 20:00:35.302096 29352 consensus_queue.cc:237] T d057ab88d0084efe8d2fe408a42a24b7 P 78bf5e3ff675477cb9f9c874292fa63f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:35.306386 29306 raft_consensus.cc:2454] T d057ab88d0084efe8d2fe408a42a24b7 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78bf5e3ff675477cb9f9c874292fa63f in term 2.
I20240501 20:00:35.307147 29232 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "419e173751fc4cd18b50341160a84d24"
dest_uuid: "78bf5e3ff675477cb9f9c874292fa63f"
from {username='slave'} at 127.0.0.1:42828
I20240501 20:00:35.307618 29232 raft_consensus.cc:482] T 419e173751fc4cd18b50341160a84d24 P 78bf5e3ff675477cb9f9c874292fa63f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:00:35.307893 29232 raft_consensus.cc:3037] T 419e173751fc4cd18b50341160a84d24 P 78bf5e3ff675477cb9f9c874292fa63f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:35.310820 29355 consensus_queue.cc:1035] T eed1ac06c82147e3bb681f4bcd1d8688 P 8feebcdef3ba428ba32e62016f3ed365 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:00:35.319665 29389 consensus_queue.cc:1035] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f [LEADER]: Connected to new peer: Peer: permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:00:35.332588 29352 consensus_queue.cc:1035] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:00:35.333274 29232 raft_consensus.cc:504] T 419e173751fc4cd18b50341160a84d24 P 78bf5e3ff675477cb9f9c874292fa63f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:35.339251 29158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "419e173751fc4cd18b50341160a84d24" candidate_uuid: "78bf5e3ff675477cb9f9c874292fa63f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8feebcdef3ba428ba32e62016f3ed365"
I20240501 20:00:35.339396 29305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "419e173751fc4cd18b50341160a84d24" candidate_uuid: "78bf5e3ff675477cb9f9c874292fa63f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "16f5b5441d8c4a44a9c0a6cca8356822"
I20240501 20:00:35.339880 29158 raft_consensus.cc:3032] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:00:35.339779 29232 leader_election.cc:288] T 419e173751fc4cd18b50341160a84d24 P 78bf5e3ff675477cb9f9c874292fa63f [CANDIDATE]: Term 2 election: Requested vote from peers 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193:43937), 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195:37025)
I20240501 20:00:35.340391 29158 raft_consensus.cc:729] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8feebcdef3ba428ba32e62016f3ed365, State: Running, Role: LEADER
I20240501 20:00:35.340422 29305 raft_consensus.cc:3037] T 419e173751fc4cd18b50341160a84d24 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:35.341205 29158 consensus_queue.cc:260] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:35.342535 29158 raft_consensus.cc:3037] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:35.342016 29056 catalog_manager.cc:5484] T 27c72ce2560f4d3baf61f0441f5c4129 P 78bf5e3ff675477cb9f9c874292fa63f reported cstate change: term changed from 1 to 2, leader changed from 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195) to 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194). New cstate: current_term: 2 leader_uuid: "78bf5e3ff675477cb9f9c874292fa63f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } health_report { overall_health: UNKNOWN } } }
I20240501 20:00:35.346305 29056 catalog_manager.cc:5484] T d057ab88d0084efe8d2fe408a42a24b7 P 78bf5e3ff675477cb9f9c874292fa63f reported cstate change: term changed from 1 to 2, leader changed from 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193) to 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194). New cstate: current_term: 2 leader_uuid: "78bf5e3ff675477cb9f9c874292fa63f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } health_report { overall_health: UNKNOWN } } }
I20240501 20:00:35.348323 29305 raft_consensus.cc:2454] T 419e173751fc4cd18b50341160a84d24 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78bf5e3ff675477cb9f9c874292fa63f in term 2.
I20240501 20:00:35.352705 29382 raft_consensus.cc:979] T eed1ac06c82147e3bb681f4bcd1d8688 P 8feebcdef3ba428ba32e62016f3ed365: : Instructing follower 78bf5e3ff675477cb9f9c874292fa63f to start an election
I20240501 20:00:35.353147 29382 raft_consensus.cc:1067] T eed1ac06c82147e3bb681f4bcd1d8688 P 8feebcdef3ba428ba32e62016f3ed365 [term 1 LEADER]: Signalling peer 78bf5e3ff675477cb9f9c874292fa63f to start an election
I20240501 20:00:35.355022 29232 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eed1ac06c82147e3bb681f4bcd1d8688"
dest_uuid: "78bf5e3ff675477cb9f9c874292fa63f"
from {username='slave'} at 127.0.0.1:42828
I20240501 20:00:35.355595 29232 raft_consensus.cc:482] T eed1ac06c82147e3bb681f4bcd1d8688 P 78bf5e3ff675477cb9f9c874292fa63f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:00:35.355960 29232 raft_consensus.cc:3037] T eed1ac06c82147e3bb681f4bcd1d8688 P 78bf5e3ff675477cb9f9c874292fa63f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:35.349278 29158 raft_consensus.cc:2454] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78bf5e3ff675477cb9f9c874292fa63f in term 2.
I20240501 20:00:35.360651 29199 leader_election.cc:302] T 419e173751fc4cd18b50341160a84d24 P 78bf5e3ff675477cb9f9c874292fa63f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78bf5e3ff675477cb9f9c874292fa63f, 8feebcdef3ba428ba32e62016f3ed365; no voters:
I20240501 20:00:35.362874 29391 raft_consensus.cc:2781] T 419e173751fc4cd18b50341160a84d24 P 78bf5e3ff675477cb9f9c874292fa63f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:00:35.363332 29352 raft_consensus.cc:979] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f: : Instructing follower 8feebcdef3ba428ba32e62016f3ed365 to start an election
I20240501 20:00:35.363394 29391 raft_consensus.cc:686] T 419e173751fc4cd18b50341160a84d24 P 78bf5e3ff675477cb9f9c874292fa63f [term 2 LEADER]: Becoming Leader. State: Replica: 78bf5e3ff675477cb9f9c874292fa63f, State: Running, Role: LEADER
I20240501 20:00:35.364092 29352 raft_consensus.cc:1067] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f [term 1 LEADER]: Signalling peer 8feebcdef3ba428ba32e62016f3ed365 to start an election
I20240501 20:00:35.364504 29391 consensus_queue.cc:237] T 419e173751fc4cd18b50341160a84d24 P 78bf5e3ff675477cb9f9c874292fa63f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:35.365659 29158 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5bd59b3e54144bd2986538a355cadd9b"
dest_uuid: "8feebcdef3ba428ba32e62016f3ed365"
from {username='slave'} at 127.0.0.1:51380
I20240501 20:00:35.365233 29232 raft_consensus.cc:504] T eed1ac06c82147e3bb681f4bcd1d8688 P 78bf5e3ff675477cb9f9c874292fa63f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:35.366273 29158 raft_consensus.cc:482] T 5bd59b3e54144bd2986538a355cadd9b P 8feebcdef3ba428ba32e62016f3ed365 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:00:35.366684 29158 raft_consensus.cc:3037] T 5bd59b3e54144bd2986538a355cadd9b P 8feebcdef3ba428ba32e62016f3ed365 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:35.370780 29156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eed1ac06c82147e3bb681f4bcd1d8688" candidate_uuid: "78bf5e3ff675477cb9f9c874292fa63f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8feebcdef3ba428ba32e62016f3ed365"
I20240501 20:00:35.371513 29156 raft_consensus.cc:3032] T eed1ac06c82147e3bb681f4bcd1d8688 P 8feebcdef3ba428ba32e62016f3ed365 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:00:35.371843 29156 raft_consensus.cc:729] T eed1ac06c82147e3bb681f4bcd1d8688 P 8feebcdef3ba428ba32e62016f3ed365 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8feebcdef3ba428ba32e62016f3ed365, State: Running, Role: LEADER
I20240501 20:00:35.372620 29156 consensus_queue.cc:260] T eed1ac06c82147e3bb681f4bcd1d8688 P 8feebcdef3ba428ba32e62016f3ed365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:35.373477 29158 raft_consensus.cc:504] T 5bd59b3e54144bd2986538a355cadd9b P 8feebcdef3ba428ba32e62016f3ed365 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:35.373850 29156 raft_consensus.cc:3037] T eed1ac06c82147e3bb681f4bcd1d8688 P 8feebcdef3ba428ba32e62016f3ed365 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:35.374706 29232 leader_election.cc:288] T eed1ac06c82147e3bb681f4bcd1d8688 P 78bf5e3ff675477cb9f9c874292fa63f [CANDIDATE]: Term 2 election: Requested vote from peers 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193:43937), 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195:37025)
I20240501 20:00:35.376482 29158 leader_election.cc:288] T 5bd59b3e54144bd2986538a355cadd9b P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 2 election: Requested vote from peers 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194:39263), 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195:37025)
I20240501 20:00:35.378046 29305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bd59b3e54144bd2986538a355cadd9b" candidate_uuid: "8feebcdef3ba428ba32e62016f3ed365" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "16f5b5441d8c4a44a9c0a6cca8356822"
I20240501 20:00:35.378262 29304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eed1ac06c82147e3bb681f4bcd1d8688" candidate_uuid: "78bf5e3ff675477cb9f9c874292fa63f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "16f5b5441d8c4a44a9c0a6cca8356822"
I20240501 20:00:35.378703 29305 raft_consensus.cc:3037] T 5bd59b3e54144bd2986538a355cadd9b P 16f5b5441d8c4a44a9c0a6cca8356822 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:35.378887 29304 raft_consensus.cc:3037] T eed1ac06c82147e3bb681f4bcd1d8688 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:35.384948 29156 raft_consensus.cc:2454] T eed1ac06c82147e3bb681f4bcd1d8688 P 8feebcdef3ba428ba32e62016f3ed365 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78bf5e3ff675477cb9f9c874292fa63f in term 2.
I20240501 20:00:35.386102 29305 raft_consensus.cc:2454] T 5bd59b3e54144bd2986538a355cadd9b P 16f5b5441d8c4a44a9c0a6cca8356822 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8feebcdef3ba428ba32e62016f3ed365 in term 2.
I20240501 20:00:35.386413 29199 leader_election.cc:302] T eed1ac06c82147e3bb681f4bcd1d8688 P 78bf5e3ff675477cb9f9c874292fa63f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78bf5e3ff675477cb9f9c874292fa63f, 8feebcdef3ba428ba32e62016f3ed365; no voters:
I20240501 20:00:35.387179 29122 leader_election.cc:302] T 5bd59b3e54144bd2986538a355cadd9b P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16f5b5441d8c4a44a9c0a6cca8356822, 8feebcdef3ba428ba32e62016f3ed365; no voters:
I20240501 20:00:35.388121 29391 raft_consensus.cc:2781] T eed1ac06c82147e3bb681f4bcd1d8688 P 78bf5e3ff675477cb9f9c874292fa63f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:00:35.388412 29232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bd59b3e54144bd2986538a355cadd9b" candidate_uuid: "8feebcdef3ba428ba32e62016f3ed365" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "78bf5e3ff675477cb9f9c874292fa63f"
I20240501 20:00:35.388638 29391 raft_consensus.cc:686] T eed1ac06c82147e3bb681f4bcd1d8688 P 78bf5e3ff675477cb9f9c874292fa63f [term 2 LEADER]: Becoming Leader. State: Replica: 78bf5e3ff675477cb9f9c874292fa63f, State: Running, Role: LEADER
I20240501 20:00:35.389036 29232 raft_consensus.cc:3032] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:00:35.389365 29354 raft_consensus.cc:979] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822: : Instructing follower 8feebcdef3ba428ba32e62016f3ed365 to start an election
I20240501 20:00:35.390017 29354 raft_consensus.cc:1067] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 1 LEADER]: Signalling peer 8feebcdef3ba428ba32e62016f3ed365 to start an election
I20240501 20:00:35.389510 29391 consensus_queue.cc:237] T eed1ac06c82147e3bb681f4bcd1d8688 P 78bf5e3ff675477cb9f9c874292fa63f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:35.389387 29054 catalog_manager.cc:5484] T 419e173751fc4cd18b50341160a84d24 P 78bf5e3ff675477cb9f9c874292fa63f reported cstate change: term changed from 1 to 2, leader changed from 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193) to 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194). New cstate: current_term: 2 leader_uuid: "78bf5e3ff675477cb9f9c874292fa63f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } health_report { overall_health: UNKNOWN } } }
I20240501 20:00:35.391862 29156 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f4895041b0924276aa587adbb8dc9e98"
dest_uuid: "8feebcdef3ba428ba32e62016f3ed365"
from {username='slave'} at 127.0.0.1:51370
I20240501 20:00:35.392568 29156 raft_consensus.cc:482] T f4895041b0924276aa587adbb8dc9e98 P 8feebcdef3ba428ba32e62016f3ed365 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:00:35.392916 29156 raft_consensus.cc:3037] T f4895041b0924276aa587adbb8dc9e98 P 8feebcdef3ba428ba32e62016f3ed365 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:35.393908 29406 raft_consensus.cc:2781] T 5bd59b3e54144bd2986538a355cadd9b P 8feebcdef3ba428ba32e62016f3ed365 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:00:35.394399 29406 raft_consensus.cc:686] T 5bd59b3e54144bd2986538a355cadd9b P 8feebcdef3ba428ba32e62016f3ed365 [term 2 LEADER]: Becoming Leader. State: Replica: 8feebcdef3ba428ba32e62016f3ed365, State: Running, Role: LEADER
I20240501 20:00:35.395172 29406 consensus_queue.cc:237] T 5bd59b3e54144bd2986538a355cadd9b P 8feebcdef3ba428ba32e62016f3ed365 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:35.387174 29304 raft_consensus.cc:2454] T eed1ac06c82147e3bb681f4bcd1d8688 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78bf5e3ff675477cb9f9c874292fa63f in term 2.
I20240501 20:00:35.389358 29232 raft_consensus.cc:729] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 78bf5e3ff675477cb9f9c874292fa63f, State: Running, Role: LEADER
I20240501 20:00:35.400216 29232 consensus_queue.cc:260] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:35.401494 29232 raft_consensus.cc:3037] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:35.403013 29156 raft_consensus.cc:504] T f4895041b0924276aa587adbb8dc9e98 P 8feebcdef3ba428ba32e62016f3ed365 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:35.403299 29054 catalog_manager.cc:5484] T 5bd59b3e54144bd2986538a355cadd9b P 8feebcdef3ba428ba32e62016f3ed365 reported cstate change: term changed from 1 to 2, leader changed from 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194) to 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193). New cstate: current_term: 2 leader_uuid: "8feebcdef3ba428ba32e62016f3ed365" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } health_report { overall_health: UNKNOWN } } }
I20240501 20:00:35.404894 29055 catalog_manager.cc:5484] T eed1ac06c82147e3bb681f4bcd1d8688 P 78bf5e3ff675477cb9f9c874292fa63f reported cstate change: term changed from 1 to 2, leader changed from 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193) to 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194). New cstate: current_term: 2 leader_uuid: "78bf5e3ff675477cb9f9c874292fa63f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } health_report { overall_health: UNKNOWN } } }
I20240501 20:00:35.405690 29233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4895041b0924276aa587adbb8dc9e98" candidate_uuid: "8feebcdef3ba428ba32e62016f3ed365" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "78bf5e3ff675477cb9f9c874292fa63f"
I20240501 20:00:35.406273 29233 raft_consensus.cc:3037] T f4895041b0924276aa587adbb8dc9e98 P 78bf5e3ff675477cb9f9c874292fa63f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:35.407907 29232 raft_consensus.cc:2454] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8feebcdef3ba428ba32e62016f3ed365 in term 2.
I20240501 20:00:35.410373 29156 leader_election.cc:288] T f4895041b0924276aa587adbb8dc9e98 P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 2 election: Requested vote from peers 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194:39263), 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195:37025)
I20240501 20:00:35.411564 29305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4895041b0924276aa587adbb8dc9e98" candidate_uuid: "8feebcdef3ba428ba32e62016f3ed365" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "16f5b5441d8c4a44a9c0a6cca8356822"
I20240501 20:00:35.412190 29305 raft_consensus.cc:3032] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:00:35.412514 29305 raft_consensus.cc:729] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 16f5b5441d8c4a44a9c0a6cca8356822, State: Running, Role: LEADER
I20240501 20:00:35.413004 29233 raft_consensus.cc:2454] T f4895041b0924276aa587adbb8dc9e98 P 78bf5e3ff675477cb9f9c874292fa63f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8feebcdef3ba428ba32e62016f3ed365 in term 2.
I20240501 20:00:35.413301 29305 consensus_queue.cc:260] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:35.414319 29124 leader_election.cc:302] T f4895041b0924276aa587adbb8dc9e98 P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78bf5e3ff675477cb9f9c874292fa63f, 8feebcdef3ba428ba32e62016f3ed365; no voters:
I20240501 20:00:35.414799 29305 raft_consensus.cc:3037] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:35.415163 29406 raft_consensus.cc:2781] T f4895041b0924276aa587adbb8dc9e98 P 8feebcdef3ba428ba32e62016f3ed365 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:00:35.415614 29406 raft_consensus.cc:686] T f4895041b0924276aa587adbb8dc9e98 P 8feebcdef3ba428ba32e62016f3ed365 [term 2 LEADER]: Becoming Leader. State: Replica: 8feebcdef3ba428ba32e62016f3ed365, State: Running, Role: LEADER
I20240501 20:00:35.416441 29406 consensus_queue.cc:237] T f4895041b0924276aa587adbb8dc9e98 P 8feebcdef3ba428ba32e62016f3ed365 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:35.420226 29305 raft_consensus.cc:2454] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8feebcdef3ba428ba32e62016f3ed365 in term 2.
I20240501 20:00:35.422420 29055 catalog_manager.cc:5484] T f4895041b0924276aa587adbb8dc9e98 P 8feebcdef3ba428ba32e62016f3ed365 reported cstate change: term changed from 1 to 2, leader changed from 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195) to 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193). New cstate: current_term: 2 leader_uuid: "8feebcdef3ba428ba32e62016f3ed365" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } health_report { overall_health: UNKNOWN } } }
I20240501 20:00:35.465756 29354 raft_consensus.cc:979] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822: : Instructing follower 8feebcdef3ba428ba32e62016f3ed365 to start an election
I20240501 20:00:35.466197 29372 raft_consensus.cc:1067] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 [term 1 LEADER]: Signalling peer 8feebcdef3ba428ba32e62016f3ed365 to start an election
I20240501 20:00:35.467458 29156 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3d51e8d55df849e5ac6e5acdc42ac63c"
dest_uuid: "8feebcdef3ba428ba32e62016f3ed365"
from {username='slave'} at 127.0.0.1:51370
I20240501 20:00:35.468029 29156 raft_consensus.cc:482] T 3d51e8d55df849e5ac6e5acdc42ac63c P 8feebcdef3ba428ba32e62016f3ed365 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:00:35.468456 29156 raft_consensus.cc:3037] T 3d51e8d55df849e5ac6e5acdc42ac63c P 8feebcdef3ba428ba32e62016f3ed365 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:35.474648 29156 raft_consensus.cc:504] T 3d51e8d55df849e5ac6e5acdc42ac63c P 8feebcdef3ba428ba32e62016f3ed365 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:35.476135 29156 leader_election.cc:288] T 3d51e8d55df849e5ac6e5acdc42ac63c P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 2 election: Requested vote from peers 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194:39263), 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195:37025)
I20240501 20:00:35.476998 29233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d51e8d55df849e5ac6e5acdc42ac63c" candidate_uuid: "8feebcdef3ba428ba32e62016f3ed365" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "78bf5e3ff675477cb9f9c874292fa63f"
I20240501 20:00:35.477057 29305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d51e8d55df849e5ac6e5acdc42ac63c" candidate_uuid: "8feebcdef3ba428ba32e62016f3ed365" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "16f5b5441d8c4a44a9c0a6cca8356822"
I20240501 20:00:35.477586 29233 raft_consensus.cc:3037] T 3d51e8d55df849e5ac6e5acdc42ac63c P 78bf5e3ff675477cb9f9c874292fa63f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:35.477622 29305 raft_consensus.cc:3032] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:00:35.477982 29305 raft_consensus.cc:729] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 16f5b5441d8c4a44a9c0a6cca8356822, State: Running, Role: LEADER
I20240501 20:00:35.478571 29305 consensus_queue.cc:260] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:35.479447 29305 raft_consensus.cc:3037] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:35.482020 29233 raft_consensus.cc:2454] T 3d51e8d55df849e5ac6e5acdc42ac63c P 78bf5e3ff675477cb9f9c874292fa63f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8feebcdef3ba428ba32e62016f3ed365 in term 2.
I20240501 20:00:35.482877 29124 leader_election.cc:302] T 3d51e8d55df849e5ac6e5acdc42ac63c P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78bf5e3ff675477cb9f9c874292fa63f, 8feebcdef3ba428ba32e62016f3ed365; no voters:
I20240501 20:00:35.483489 29406 raft_consensus.cc:2781] T 3d51e8d55df849e5ac6e5acdc42ac63c P 8feebcdef3ba428ba32e62016f3ed365 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:00:35.483778 29305 raft_consensus.cc:2454] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8feebcdef3ba428ba32e62016f3ed365 in term 2.
I20240501 20:00:35.483832 29406 raft_consensus.cc:686] T 3d51e8d55df849e5ac6e5acdc42ac63c P 8feebcdef3ba428ba32e62016f3ed365 [term 2 LEADER]: Becoming Leader. State: Replica: 8feebcdef3ba428ba32e62016f3ed365, State: Running, Role: LEADER
I20240501 20:00:35.484817 29406 consensus_queue.cc:237] T 3d51e8d55df849e5ac6e5acdc42ac63c P 8feebcdef3ba428ba32e62016f3ed365 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:35.490479 29055 catalog_manager.cc:5484] T 3d51e8d55df849e5ac6e5acdc42ac63c P 8feebcdef3ba428ba32e62016f3ed365 reported cstate change: term changed from 1 to 2, leader changed from 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195) to 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193). New cstate: current_term: 2 leader_uuid: "8feebcdef3ba428ba32e62016f3ed365" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } health_report { overall_health: UNKNOWN } } }
I20240501 20:00:35.574810 29354 raft_consensus.cc:979] T afbcfea6a7894b5ab7c122d3d4134869 P 16f5b5441d8c4a44a9c0a6cca8356822: : Instructing follower 8feebcdef3ba428ba32e62016f3ed365 to start an election
I20240501 20:00:35.575193 29372 raft_consensus.cc:1067] T afbcfea6a7894b5ab7c122d3d4134869 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 1 LEADER]: Signalling peer 8feebcdef3ba428ba32e62016f3ed365 to start an election
I20240501 20:00:35.576476 29156 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "afbcfea6a7894b5ab7c122d3d4134869"
dest_uuid: "8feebcdef3ba428ba32e62016f3ed365"
from {username='slave'} at 127.0.0.1:51370
I20240501 20:00:35.576928 29156 raft_consensus.cc:482] T afbcfea6a7894b5ab7c122d3d4134869 P 8feebcdef3ba428ba32e62016f3ed365 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:00:35.577208 29156 raft_consensus.cc:3037] T afbcfea6a7894b5ab7c122d3d4134869 P 8feebcdef3ba428ba32e62016f3ed365 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:35.581516 29156 raft_consensus.cc:504] T afbcfea6a7894b5ab7c122d3d4134869 P 8feebcdef3ba428ba32e62016f3ed365 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:35.582963 29156 leader_election.cc:288] T afbcfea6a7894b5ab7c122d3d4134869 P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 2 election: Requested vote from peers 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194:39263), 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195:37025)
I20240501 20:00:35.583792 29233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afbcfea6a7894b5ab7c122d3d4134869" candidate_uuid: "8feebcdef3ba428ba32e62016f3ed365" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "78bf5e3ff675477cb9f9c874292fa63f"
I20240501 20:00:35.583901 29305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afbcfea6a7894b5ab7c122d3d4134869" candidate_uuid: "8feebcdef3ba428ba32e62016f3ed365" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "16f5b5441d8c4a44a9c0a6cca8356822"
I20240501 20:00:35.584395 29233 raft_consensus.cc:3037] T afbcfea6a7894b5ab7c122d3d4134869 P 78bf5e3ff675477cb9f9c874292fa63f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:35.584460 29305 raft_consensus.cc:3032] T afbcfea6a7894b5ab7c122d3d4134869 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:00:35.584787 29305 raft_consensus.cc:729] T afbcfea6a7894b5ab7c122d3d4134869 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 16f5b5441d8c4a44a9c0a6cca8356822, State: Running, Role: LEADER
I20240501 20:00:35.585397 29305 consensus_queue.cc:260] T afbcfea6a7894b5ab7c122d3d4134869 P 16f5b5441d8c4a44a9c0a6cca8356822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:35.586263 29305 raft_consensus.cc:3037] T afbcfea6a7894b5ab7c122d3d4134869 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:35.588694 29233 raft_consensus.cc:2454] T afbcfea6a7894b5ab7c122d3d4134869 P 78bf5e3ff675477cb9f9c874292fa63f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8feebcdef3ba428ba32e62016f3ed365 in term 2.
I20240501 20:00:35.589534 29124 leader_election.cc:302] T afbcfea6a7894b5ab7c122d3d4134869 P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78bf5e3ff675477cb9f9c874292fa63f, 8feebcdef3ba428ba32e62016f3ed365; no voters:
I20240501 20:00:35.590206 29406 raft_consensus.cc:2781] T afbcfea6a7894b5ab7c122d3d4134869 P 8feebcdef3ba428ba32e62016f3ed365 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:00:35.590572 29406 raft_consensus.cc:686] T afbcfea6a7894b5ab7c122d3d4134869 P 8feebcdef3ba428ba32e62016f3ed365 [term 2 LEADER]: Becoming Leader. State: Replica: 8feebcdef3ba428ba32e62016f3ed365, State: Running, Role: LEADER
I20240501 20:00:35.590672 29305 raft_consensus.cc:2454] T afbcfea6a7894b5ab7c122d3d4134869 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8feebcdef3ba428ba32e62016f3ed365 in term 2.
I20240501 20:00:35.591359 29406 consensus_queue.cc:237] T afbcfea6a7894b5ab7c122d3d4134869 P 8feebcdef3ba428ba32e62016f3ed365 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:35.597225 29055 catalog_manager.cc:5484] T afbcfea6a7894b5ab7c122d3d4134869 P 8feebcdef3ba428ba32e62016f3ed365 reported cstate change: term changed from 1 to 2, leader changed from 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195) to 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193). New cstate: current_term: 2 leader_uuid: "8feebcdef3ba428ba32e62016f3ed365" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } health_report { overall_health: UNKNOWN } } }
I20240501 20:00:35.602854 29352 raft_consensus.cc:979] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 78bf5e3ff675477cb9f9c874292fa63f: : Instructing follower 8feebcdef3ba428ba32e62016f3ed365 to start an election
I20240501 20:00:35.603266 29391 raft_consensus.cc:1067] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 78bf5e3ff675477cb9f9c874292fa63f [term 1 LEADER]: Signalling peer 8feebcdef3ba428ba32e62016f3ed365 to start an election
I20240501 20:00:35.604773 29156 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0d9b61e9c98e4d19b9272fe70fab0c3d"
dest_uuid: "8feebcdef3ba428ba32e62016f3ed365"
from {username='slave'} at 127.0.0.1:51380
I20240501 20:00:35.605271 29156 raft_consensus.cc:482] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 8feebcdef3ba428ba32e62016f3ed365 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:00:35.605566 29156 raft_consensus.cc:3037] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 8feebcdef3ba428ba32e62016f3ed365 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:35.609721 29156 raft_consensus.cc:504] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 8feebcdef3ba428ba32e62016f3ed365 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } } peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } }
I20240501 20:00:35.611091 29156 leader_election.cc:288] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 2 election: Requested vote from peers 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195:37025), 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194:39263)
I20240501 20:00:35.611927 29305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d9b61e9c98e4d19b9272fe70fab0c3d" candidate_uuid: "8feebcdef3ba428ba32e62016f3ed365" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "16f5b5441d8c4a44a9c0a6cca8356822"
I20240501 20:00:35.612010 29233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d9b61e9c98e4d19b9272fe70fab0c3d" candidate_uuid: "8feebcdef3ba428ba32e62016f3ed365" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "78bf5e3ff675477cb9f9c874292fa63f"
I20240501 20:00:35.612525 29305 raft_consensus.cc:3037] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 16f5b5441d8c4a44a9c0a6cca8356822 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:35.612600 29233 raft_consensus.cc:3032] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 78bf5e3ff675477cb9f9c874292fa63f [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:00:35.612936 29233 raft_consensus.cc:729] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 78bf5e3ff675477cb9f9c874292fa63f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 78bf5e3ff675477cb9f9c874292fa63f, State: Running, Role: LEADER
I20240501 20:00:35.613567 29233 consensus_queue.cc:260] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 78bf5e3ff675477cb9f9c874292fa63f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } } peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } }
I20240501 20:00:35.614490 29233 raft_consensus.cc:3037] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 78bf5e3ff675477cb9f9c874292fa63f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:00:35.617929 29305 raft_consensus.cc:2454] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 16f5b5441d8c4a44a9c0a6cca8356822 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8feebcdef3ba428ba32e62016f3ed365 in term 2.
I20240501 20:00:35.618824 29122 leader_election.cc:302] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16f5b5441d8c4a44a9c0a6cca8356822, 8feebcdef3ba428ba32e62016f3ed365; no voters:
I20240501 20:00:35.619338 29233 raft_consensus.cc:2454] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 78bf5e3ff675477cb9f9c874292fa63f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8feebcdef3ba428ba32e62016f3ed365 in term 2.
I20240501 20:00:35.619701 29406 raft_consensus.cc:2781] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 8feebcdef3ba428ba32e62016f3ed365 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:00:35.620147 29406 raft_consensus.cc:686] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 8feebcdef3ba428ba32e62016f3ed365 [term 2 LEADER]: Becoming Leader. State: Replica: 8feebcdef3ba428ba32e62016f3ed365, State: Running, Role: LEADER
I20240501 20:00:35.620887 29406 consensus_queue.cc:237] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 8feebcdef3ba428ba32e62016f3ed365 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } } peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } }
I20240501 20:00:35.626930 29055 catalog_manager.cc:5484] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 8feebcdef3ba428ba32e62016f3ed365 reported cstate change: term changed from 1 to 2, leader changed from 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194) to 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193). New cstate: current_term: 2 leader_uuid: "8feebcdef3ba428ba32e62016f3ed365" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } health_report { overall_health: UNKNOWN } } }
I20240501 20:00:35.663679 29156 raft_consensus.cc:1261] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 8feebcdef3ba428ba32e62016f3ed365 [term 2 FOLLOWER]: Refusing update from remote peer 16f5b5441d8c4a44a9c0a6cca8356822: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:00:35.664885 29372 consensus_queue.cc:1035] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 16f5b5441d8c4a44a9c0a6cca8356822 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:00:35.672245 29233 raft_consensus.cc:1261] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 78bf5e3ff675477cb9f9c874292fa63f [term 2 FOLLOWER]: Refusing update from remote peer 16f5b5441d8c4a44a9c0a6cca8356822: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:00:35.674021 29372 consensus_queue.cc:1035] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 16f5b5441d8c4a44a9c0a6cca8356822 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:00:35.682696 29156 raft_consensus.cc:1261] T 27c72ce2560f4d3baf61f0441f5c4129 P 8feebcdef3ba428ba32e62016f3ed365 [term 2 FOLLOWER]: Refusing update from remote peer 78bf5e3ff675477cb9f9c874292fa63f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:00:35.683943 29391 consensus_queue.cc:1035] T 27c72ce2560f4d3baf61f0441f5c4129 P 78bf5e3ff675477cb9f9c874292fa63f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:00:35.690623 29305 raft_consensus.cc:1261] T 27c72ce2560f4d3baf61f0441f5c4129 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 2 FOLLOWER]: Refusing update from remote peer 78bf5e3ff675477cb9f9c874292fa63f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:00:35.692226 29386 consensus_queue.cc:1035] T 27c72ce2560f4d3baf61f0441f5c4129 P 78bf5e3ff675477cb9f9c874292fa63f [LEADER]: Connected to new peer: Peer: permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:00:35.790983 29305 raft_consensus.cc:1261] T 419e173751fc4cd18b50341160a84d24 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 2 FOLLOWER]: Refusing update from remote peer 78bf5e3ff675477cb9f9c874292fa63f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:00:35.792207 29386 consensus_queue.cc:1035] T 419e173751fc4cd18b50341160a84d24 P 78bf5e3ff675477cb9f9c874292fa63f [LEADER]: Connected to new peer: Peer: permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:00:35.799559 29156 raft_consensus.cc:1261] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 [term 2 FOLLOWER]: Refusing update from remote peer 78bf5e3ff675477cb9f9c874292fa63f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:00:35.800957 29386 consensus_queue.cc:1035] T 419e173751fc4cd18b50341160a84d24 P 78bf5e3ff675477cb9f9c874292fa63f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:00:35.834666 29305 raft_consensus.cc:1261] T d057ab88d0084efe8d2fe408a42a24b7 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 2 FOLLOWER]: Refusing update from remote peer 78bf5e3ff675477cb9f9c874292fa63f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:00:35.835816 29386 consensus_queue.cc:1035] T d057ab88d0084efe8d2fe408a42a24b7 P 78bf5e3ff675477cb9f9c874292fa63f [LEADER]: Connected to new peer: Peer: permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:00:35.842979 29156 raft_consensus.cc:1261] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365 [term 2 FOLLOWER]: Refusing update from remote peer 78bf5e3ff675477cb9f9c874292fa63f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:00:35.844416 29352 consensus_queue.cc:1035] T d057ab88d0084efe8d2fe408a42a24b7 P 78bf5e3ff675477cb9f9c874292fa63f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:00:35.876331 29233 raft_consensus.cc:1261] T 3d51e8d55df849e5ac6e5acdc42ac63c P 78bf5e3ff675477cb9f9c874292fa63f [term 2 FOLLOWER]: Refusing update from remote peer 8feebcdef3ba428ba32e62016f3ed365: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:00:35.877540 29406 consensus_queue.cc:1035] T 3d51e8d55df849e5ac6e5acdc42ac63c P 8feebcdef3ba428ba32e62016f3ed365 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:00:35.885280 29305 raft_consensus.cc:1261] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 [term 2 FOLLOWER]: Refusing update from remote peer 8feebcdef3ba428ba32e62016f3ed365: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:00:35.886710 29398 consensus_queue.cc:1035] T 3d51e8d55df849e5ac6e5acdc42ac63c P 8feebcdef3ba428ba32e62016f3ed365 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:00:35.892288 29156 raft_consensus.cc:1261] T eed1ac06c82147e3bb681f4bcd1d8688 P 8feebcdef3ba428ba32e62016f3ed365 [term 2 FOLLOWER]: Refusing update from remote peer 78bf5e3ff675477cb9f9c874292fa63f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:00:35.893460 29352 consensus_queue.cc:1035] T eed1ac06c82147e3bb681f4bcd1d8688 P 78bf5e3ff675477cb9f9c874292fa63f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:00:35.900152 29305 raft_consensus.cc:1261] T eed1ac06c82147e3bb681f4bcd1d8688 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 2 FOLLOWER]: Refusing update from remote peer 78bf5e3ff675477cb9f9c874292fa63f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:00:35.901609 29386 consensus_queue.cc:1035] T eed1ac06c82147e3bb681f4bcd1d8688 P 78bf5e3ff675477cb9f9c874292fa63f [LEADER]: Connected to new peer: Peer: permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:00:35.945446 29305 raft_consensus.cc:1261] T 5bd59b3e54144bd2986538a355cadd9b P 16f5b5441d8c4a44a9c0a6cca8356822 [term 2 FOLLOWER]: Refusing update from remote peer 8feebcdef3ba428ba32e62016f3ed365: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:00:35.946492 29398 consensus_queue.cc:1035] T 5bd59b3e54144bd2986538a355cadd9b P 8feebcdef3ba428ba32e62016f3ed365 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:00:35.953682 29233 raft_consensus.cc:1261] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f [term 2 FOLLOWER]: Refusing update from remote peer 8feebcdef3ba428ba32e62016f3ed365: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:00:35.954986 29406 consensus_queue.cc:1035] T 5bd59b3e54144bd2986538a355cadd9b P 8feebcdef3ba428ba32e62016f3ed365 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:00:35.990422 29233 raft_consensus.cc:1261] T f4895041b0924276aa587adbb8dc9e98 P 78bf5e3ff675477cb9f9c874292fa63f [term 2 FOLLOWER]: Refusing update from remote peer 8feebcdef3ba428ba32e62016f3ed365: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:00:35.991593 29406 consensus_queue.cc:1035] T f4895041b0924276aa587adbb8dc9e98 P 8feebcdef3ba428ba32e62016f3ed365 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:00:36.001274 29305 raft_consensus.cc:1261] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 2 FOLLOWER]: Refusing update from remote peer 8feebcdef3ba428ba32e62016f3ed365: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:00:36.002607 29440 consensus_queue.cc:1035] T f4895041b0924276aa587adbb8dc9e98 P 8feebcdef3ba428ba32e62016f3ed365 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:00:36.019111 29305 raft_consensus.cc:1261] T afbcfea6a7894b5ab7c122d3d4134869 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 2 FOLLOWER]: Refusing update from remote peer 8feebcdef3ba428ba32e62016f3ed365: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:00:36.020181 29440 consensus_queue.cc:1035] T afbcfea6a7894b5ab7c122d3d4134869 P 8feebcdef3ba428ba32e62016f3ed365 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:00:36.026629 29233 raft_consensus.cc:1261] T afbcfea6a7894b5ab7c122d3d4134869 P 78bf5e3ff675477cb9f9c874292fa63f [term 2 FOLLOWER]: Refusing update from remote peer 8feebcdef3ba428ba32e62016f3ed365: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:00:36.027981 29440 consensus_queue.cc:1035] T afbcfea6a7894b5ab7c122d3d4134869 P 8feebcdef3ba428ba32e62016f3ed365 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:00:36.143808 29233 raft_consensus.cc:1261] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 78bf5e3ff675477cb9f9c874292fa63f [term 2 FOLLOWER]: Refusing update from remote peer 8feebcdef3ba428ba32e62016f3ed365: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:00:36.145160 29440 consensus_queue.cc:1035] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 8feebcdef3ba428ba32e62016f3ed365 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:00:36.152928 29305 raft_consensus.cc:1261] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 16f5b5441d8c4a44a9c0a6cca8356822 [term 2 FOLLOWER]: Refusing update from remote peer 8feebcdef3ba428ba32e62016f3ed365: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:00:36.155269 29440 consensus_queue.cc:1035] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 8feebcdef3ba428ba32e62016f3ed365 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:00:38.173730 28519 auto_leader_rebalancer-test.cc:331] The leader distribution is
I20240501 20:00:38.174466 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:38.195401 29158 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3d51e8d55df849e5ac6e5acdc42ac63c"
dest_uuid: "8feebcdef3ba428ba32e62016f3ed365"
mode: GRACEFUL
new_leader_uuid: "16f5b5441d8c4a44a9c0a6cca8356822"
from {username='slave'} at 127.0.0.1:48854
I20240501 20:00:38.195919 29158 raft_consensus.cc:595] T 3d51e8d55df849e5ac6e5acdc42ac63c P 8feebcdef3ba428ba32e62016f3ed365 [term 2 LEADER]: Received request to transfer leadership to TS 16f5b5441d8c4a44a9c0a6cca8356822
I20240501 20:00:38.198035 29158 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f4895041b0924276aa587adbb8dc9e98"
dest_uuid: "8feebcdef3ba428ba32e62016f3ed365"
mode: GRACEFUL
new_leader_uuid: "16f5b5441d8c4a44a9c0a6cca8356822"
from {username='slave'} at 127.0.0.1:48854
I20240501 20:00:38.198482 29158 raft_consensus.cc:595] T f4895041b0924276aa587adbb8dc9e98 P 8feebcdef3ba428ba32e62016f3ed365 [term 2 LEADER]: Received request to transfer leadership to TS 16f5b5441d8c4a44a9c0a6cca8356822
I20240501 20:00:38.199291 28519 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240501 20:00:38.199708 28519 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:38.390547 29450 raft_consensus.cc:979] T 3d51e8d55df849e5ac6e5acdc42ac63c P 8feebcdef3ba428ba32e62016f3ed365: : Instructing follower 16f5b5441d8c4a44a9c0a6cca8356822 to start an election
I20240501 20:00:38.390985 29406 raft_consensus.cc:1067] T 3d51e8d55df849e5ac6e5acdc42ac63c P 8feebcdef3ba428ba32e62016f3ed365 [term 2 LEADER]: Signalling peer 16f5b5441d8c4a44a9c0a6cca8356822 to start an election
I20240501 20:00:38.392403 29305 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3d51e8d55df849e5ac6e5acdc42ac63c"
dest_uuid: "16f5b5441d8c4a44a9c0a6cca8356822"
from {username='slave'} at 127.0.0.1:39788
I20240501 20:00:38.392987 29305 raft_consensus.cc:482] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:00:38.393376 29305 raft_consensus.cc:3037] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:00:38.397732 29305 raft_consensus.cc:504] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:38.399156 29305 leader_election.cc:288] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 [CANDIDATE]: Term 3 election: Requested vote from peers 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193:43937), 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194:39263)
I20240501 20:00:38.400024 29158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d51e8d55df849e5ac6e5acdc42ac63c" candidate_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8feebcdef3ba428ba32e62016f3ed365"
I20240501 20:00:38.400142 29233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d51e8d55df849e5ac6e5acdc42ac63c" candidate_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "78bf5e3ff675477cb9f9c874292fa63f"
I20240501 20:00:38.400645 29158 raft_consensus.cc:3032] T 3d51e8d55df849e5ac6e5acdc42ac63c P 8feebcdef3ba428ba32e62016f3ed365 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:00:38.400713 29233 raft_consensus.cc:3037] T 3d51e8d55df849e5ac6e5acdc42ac63c P 78bf5e3ff675477cb9f9c874292fa63f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:00:38.400975 29158 raft_consensus.cc:729] T 3d51e8d55df849e5ac6e5acdc42ac63c P 8feebcdef3ba428ba32e62016f3ed365 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8feebcdef3ba428ba32e62016f3ed365, State: Running, Role: LEADER
I20240501 20:00:38.401633 29158 consensus_queue.cc:260] T 3d51e8d55df849e5ac6e5acdc42ac63c P 8feebcdef3ba428ba32e62016f3ed365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:38.402523 29158 raft_consensus.cc:3037] T 3d51e8d55df849e5ac6e5acdc42ac63c P 8feebcdef3ba428ba32e62016f3ed365 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:00:38.404672 29233 raft_consensus.cc:2454] T 3d51e8d55df849e5ac6e5acdc42ac63c P 78bf5e3ff675477cb9f9c874292fa63f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16f5b5441d8c4a44a9c0a6cca8356822 in term 3.
I20240501 20:00:38.405532 29273 leader_election.cc:302] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16f5b5441d8c4a44a9c0a6cca8356822, 78bf5e3ff675477cb9f9c874292fa63f; no voters:
I20240501 20:00:38.406184 29372 raft_consensus.cc:2781] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:00:38.407315 29158 raft_consensus.cc:2454] T 3d51e8d55df849e5ac6e5acdc42ac63c P 8feebcdef3ba428ba32e62016f3ed365 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16f5b5441d8c4a44a9c0a6cca8356822 in term 3.
I20240501 20:00:38.407496 29372 raft_consensus.cc:686] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 [term 3 LEADER]: Becoming Leader. State: Replica: 16f5b5441d8c4a44a9c0a6cca8356822, State: Running, Role: LEADER
I20240501 20:00:38.408715 29372 consensus_queue.cc:237] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:38.415163 29054 catalog_manager.cc:5484] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 reported cstate change: term changed from 2 to 3, leader changed from 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193) to 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195). New cstate: current_term: 3 leader_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } health_report { overall_health: HEALTHY } } }
I20240501 20:00:38.477900 29450 raft_consensus.cc:979] T f4895041b0924276aa587adbb8dc9e98 P 8feebcdef3ba428ba32e62016f3ed365: : Instructing follower 16f5b5441d8c4a44a9c0a6cca8356822 to start an election
I20240501 20:00:38.478325 29406 raft_consensus.cc:1067] T f4895041b0924276aa587adbb8dc9e98 P 8feebcdef3ba428ba32e62016f3ed365 [term 2 LEADER]: Signalling peer 16f5b5441d8c4a44a9c0a6cca8356822 to start an election
I20240501 20:00:38.479730 29305 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f4895041b0924276aa587adbb8dc9e98"
dest_uuid: "16f5b5441d8c4a44a9c0a6cca8356822"
from {username='slave'} at 127.0.0.1:39788
I20240501 20:00:38.480211 29305 raft_consensus.cc:482] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:00:38.480496 29305 raft_consensus.cc:3037] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:00:38.484800 29305 raft_consensus.cc:504] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:38.486047 29305 leader_election.cc:288] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 [CANDIDATE]: Term 3 election: Requested vote from peers 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193:43937), 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194:39263)
I20240501 20:00:38.487000 29158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4895041b0924276aa587adbb8dc9e98" candidate_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8feebcdef3ba428ba32e62016f3ed365"
I20240501 20:00:38.487217 29233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4895041b0924276aa587adbb8dc9e98" candidate_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "78bf5e3ff675477cb9f9c874292fa63f"
I20240501 20:00:38.487607 29158 raft_consensus.cc:3032] T f4895041b0924276aa587adbb8dc9e98 P 8feebcdef3ba428ba32e62016f3ed365 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:00:38.487684 29233 raft_consensus.cc:3037] T f4895041b0924276aa587adbb8dc9e98 P 78bf5e3ff675477cb9f9c874292fa63f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:00:38.488010 29158 raft_consensus.cc:729] T f4895041b0924276aa587adbb8dc9e98 P 8feebcdef3ba428ba32e62016f3ed365 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8feebcdef3ba428ba32e62016f3ed365, State: Running, Role: LEADER
I20240501 20:00:38.488667 29158 consensus_queue.cc:260] T f4895041b0924276aa587adbb8dc9e98 P 8feebcdef3ba428ba32e62016f3ed365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:38.489490 29158 raft_consensus.cc:3037] T f4895041b0924276aa587adbb8dc9e98 P 8feebcdef3ba428ba32e62016f3ed365 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:00:38.493156 29233 raft_consensus.cc:2454] T f4895041b0924276aa587adbb8dc9e98 P 78bf5e3ff675477cb9f9c874292fa63f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16f5b5441d8c4a44a9c0a6cca8356822 in term 3.
I20240501 20:00:38.494108 29273 leader_election.cc:302] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16f5b5441d8c4a44a9c0a6cca8356822, 78bf5e3ff675477cb9f9c874292fa63f; no voters:
I20240501 20:00:38.494761 29372 raft_consensus.cc:2781] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:00:38.495110 29372 raft_consensus.cc:686] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 3 LEADER]: Becoming Leader. State: Replica: 16f5b5441d8c4a44a9c0a6cca8356822, State: Running, Role: LEADER
I20240501 20:00:38.495384 29158 raft_consensus.cc:2454] T f4895041b0924276aa587adbb8dc9e98 P 8feebcdef3ba428ba32e62016f3ed365 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16f5b5441d8c4a44a9c0a6cca8356822 in term 3.
I20240501 20:00:38.495784 29372 consensus_queue.cc:237] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:38.501740 29054 catalog_manager.cc:5484] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 reported cstate change: term changed from 2 to 3, leader changed from 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193) to 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195). New cstate: current_term: 3 leader_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } health_report { overall_health: HEALTHY } } }
I20240501 20:00:38.818579 29233 raft_consensus.cc:1261] T 3d51e8d55df849e5ac6e5acdc42ac63c P 78bf5e3ff675477cb9f9c874292fa63f [term 3 FOLLOWER]: Refusing update from remote peer 16f5b5441d8c4a44a9c0a6cca8356822: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 20:00:38.819815 29372 consensus_queue.cc:1035] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:00:38.828961 29158 raft_consensus.cc:1261] T 3d51e8d55df849e5ac6e5acdc42ac63c P 8feebcdef3ba428ba32e62016f3ed365 [term 3 FOLLOWER]: Refusing update from remote peer 16f5b5441d8c4a44a9c0a6cca8356822: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 20:00:38.830367 29451 consensus_queue.cc:1035] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:00:38.907701 29233 raft_consensus.cc:1261] T f4895041b0924276aa587adbb8dc9e98 P 78bf5e3ff675477cb9f9c874292fa63f [term 3 FOLLOWER]: Refusing update from remote peer 16f5b5441d8c4a44a9c0a6cca8356822: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 20:00:38.908843 29372 consensus_queue.cc:1035] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:00:38.918115 29158 raft_consensus.cc:1261] T f4895041b0924276aa587adbb8dc9e98 P 8feebcdef3ba428ba32e62016f3ed365 [term 3 FOLLOWER]: Refusing update from remote peer 16f5b5441d8c4a44a9c0a6cca8356822: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 20:00:38.919457 29464 consensus_queue.cc:1035] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:00:39.200601 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:39.205653 28519 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:39.206045 28519 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:40.206808 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:40.211984 28519 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:40.212395 28519 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:41.213097 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:41.218237 28519 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:41.218577 28519 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:42.219300 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:42.224141 28519 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:42.224438 28519 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:43.225433 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:43.232085 28519 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:43.232551 28519 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:44.233278 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:44.238515 28519 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:44.238927 28519 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:45.239791 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:45.245146 28519 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:45.245507 28519 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:46.246167 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:46.251066 28519 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:46.251415 28519 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:47.252100 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:47.256976 28519 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:47.257336 28519 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:48.278023 28519 auto_leader_rebalancer-test.cc:350] The leader distribution is
I20240501 20:00:48.283977 29305 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3d51e8d55df849e5ac6e5acdc42ac63c"
dest_uuid: "16f5b5441d8c4a44a9c0a6cca8356822"
mode: GRACEFUL
new_leader_uuid: "8feebcdef3ba428ba32e62016f3ed365"
from {username='slave'} at 127.0.0.1:56748
I20240501 20:00:48.284802 29305 raft_consensus.cc:595] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 [term 3 LEADER]: Received request to transfer leadership to TS 8feebcdef3ba428ba32e62016f3ed365
I20240501 20:00:48.288583 29305 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f4895041b0924276aa587adbb8dc9e98"
dest_uuid: "16f5b5441d8c4a44a9c0a6cca8356822"
mode: GRACEFUL
new_leader_uuid: "8feebcdef3ba428ba32e62016f3ed365"
from {username='slave'} at 127.0.0.1:56748
I20240501 20:00:48.288888 29305 raft_consensus.cc:595] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 3 LEADER]: Received request to transfer leadership to TS 8feebcdef3ba428ba32e62016f3ed365
I20240501 20:00:48.296208 29233 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "27c72ce2560f4d3baf61f0441f5c4129"
dest_uuid: "78bf5e3ff675477cb9f9c874292fa63f"
mode: GRACEFUL
new_leader_uuid: "8feebcdef3ba428ba32e62016f3ed365"
from {username='slave'} at 127.0.0.1:41300
I20240501 20:00:48.296586 29233 raft_consensus.cc:595] T 27c72ce2560f4d3baf61f0441f5c4129 P 78bf5e3ff675477cb9f9c874292fa63f [term 2 LEADER]: Received request to transfer leadership to TS 8feebcdef3ba428ba32e62016f3ed365
I20240501 20:00:48.300309 29233 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "419e173751fc4cd18b50341160a84d24"
dest_uuid: "78bf5e3ff675477cb9f9c874292fa63f"
mode: GRACEFUL
new_leader_uuid: "8feebcdef3ba428ba32e62016f3ed365"
from {username='slave'} at 127.0.0.1:41300
I20240501 20:00:48.300697 29233 raft_consensus.cc:595] T 419e173751fc4cd18b50341160a84d24 P 78bf5e3ff675477cb9f9c874292fa63f [term 2 LEADER]: Received request to transfer leadership to TS 8feebcdef3ba428ba32e62016f3ed365
I20240501 20:00:48.304003 29233 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d057ab88d0084efe8d2fe408a42a24b7"
dest_uuid: "78bf5e3ff675477cb9f9c874292fa63f"
mode: GRACEFUL
new_leader_uuid: "8feebcdef3ba428ba32e62016f3ed365"
from {username='slave'} at 127.0.0.1:41300
I20240501 20:00:48.304354 29233 raft_consensus.cc:595] T d057ab88d0084efe8d2fe408a42a24b7 P 78bf5e3ff675477cb9f9c874292fa63f [term 2 LEADER]: Received request to transfer leadership to TS 8feebcdef3ba428ba32e62016f3ed365
I20240501 20:00:48.391347 29471 raft_consensus.cc:979] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822: : Instructing follower 8feebcdef3ba428ba32e62016f3ed365 to start an election
I20240501 20:00:48.391722 29471 raft_consensus.cc:1067] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 [term 3 LEADER]: Signalling peer 8feebcdef3ba428ba32e62016f3ed365 to start an election
I20240501 20:00:48.392940 29158 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3d51e8d55df849e5ac6e5acdc42ac63c"
dest_uuid: "8feebcdef3ba428ba32e62016f3ed365"
from {username='slave'} at 127.0.0.1:51370
I20240501 20:00:48.393301 29158 raft_consensus.cc:482] T 3d51e8d55df849e5ac6e5acdc42ac63c P 8feebcdef3ba428ba32e62016f3ed365 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:00:48.393587 29158 raft_consensus.cc:3037] T 3d51e8d55df849e5ac6e5acdc42ac63c P 8feebcdef3ba428ba32e62016f3ed365 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:00:48.398144 29158 raft_consensus.cc:504] T 3d51e8d55df849e5ac6e5acdc42ac63c P 8feebcdef3ba428ba32e62016f3ed365 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:48.399575 29158 leader_election.cc:288] T 3d51e8d55df849e5ac6e5acdc42ac63c P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 4 election: Requested vote from peers 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194:39263), 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195:37025)
I20240501 20:00:48.400301 29233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d51e8d55df849e5ac6e5acdc42ac63c" candidate_uuid: "8feebcdef3ba428ba32e62016f3ed365" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "78bf5e3ff675477cb9f9c874292fa63f"
I20240501 20:00:48.400555 29305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d51e8d55df849e5ac6e5acdc42ac63c" candidate_uuid: "8feebcdef3ba428ba32e62016f3ed365" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "16f5b5441d8c4a44a9c0a6cca8356822"
I20240501 20:00:48.400983 29233 raft_consensus.cc:3037] T 3d51e8d55df849e5ac6e5acdc42ac63c P 78bf5e3ff675477cb9f9c874292fa63f [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:00:48.401010 29305 raft_consensus.cc:3032] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:00:48.401398 29305 raft_consensus.cc:729] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 16f5b5441d8c4a44a9c0a6cca8356822, State: Running, Role: LEADER
I20240501 20:00:48.402012 29305 consensus_queue.cc:260] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:48.402786 29305 raft_consensus.cc:3037] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:00:48.406510 29233 raft_consensus.cc:2454] T 3d51e8d55df849e5ac6e5acdc42ac63c P 78bf5e3ff675477cb9f9c874292fa63f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8feebcdef3ba428ba32e62016f3ed365 in term 4.
I20240501 20:00:48.406670 29305 raft_consensus.cc:2454] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8feebcdef3ba428ba32e62016f3ed365 in term 4.
I20240501 20:00:48.407585 29124 leader_election.cc:302] T 3d51e8d55df849e5ac6e5acdc42ac63c P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78bf5e3ff675477cb9f9c874292fa63f, 8feebcdef3ba428ba32e62016f3ed365; no voters:
I20240501 20:00:48.408315 29468 raft_consensus.cc:2781] T 3d51e8d55df849e5ac6e5acdc42ac63c P 8feebcdef3ba428ba32e62016f3ed365 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:00:48.409046 29468 raft_consensus.cc:686] T 3d51e8d55df849e5ac6e5acdc42ac63c P 8feebcdef3ba428ba32e62016f3ed365 [term 4 LEADER]: Becoming Leader. State: Replica: 8feebcdef3ba428ba32e62016f3ed365, State: Running, Role: LEADER
I20240501 20:00:48.409895 29468 consensus_queue.cc:237] T 3d51e8d55df849e5ac6e5acdc42ac63c P 8feebcdef3ba428ba32e62016f3ed365 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:48.416110 29055 catalog_manager.cc:5484] T 3d51e8d55df849e5ac6e5acdc42ac63c P 8feebcdef3ba428ba32e62016f3ed365 reported cstate change: term changed from 3 to 4, leader changed from 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195) to 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193). New cstate: current_term: 4 leader_uuid: "8feebcdef3ba428ba32e62016f3ed365" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } health_report { overall_health: UNKNOWN } } }
I20240501 20:00:48.432284 29474 raft_consensus.cc:979] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822: : Instructing follower 8feebcdef3ba428ba32e62016f3ed365 to start an election
I20240501 20:00:48.432684 29471 raft_consensus.cc:1067] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 3 LEADER]: Signalling peer 8feebcdef3ba428ba32e62016f3ed365 to start an election
I20240501 20:00:48.433908 29158 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f4895041b0924276aa587adbb8dc9e98"
dest_uuid: "8feebcdef3ba428ba32e62016f3ed365"
from {username='slave'} at 127.0.0.1:51370
I20240501 20:00:48.434312 29158 raft_consensus.cc:482] T f4895041b0924276aa587adbb8dc9e98 P 8feebcdef3ba428ba32e62016f3ed365 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:00:48.434576 29158 raft_consensus.cc:3037] T f4895041b0924276aa587adbb8dc9e98 P 8feebcdef3ba428ba32e62016f3ed365 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:00:48.439136 29158 raft_consensus.cc:504] T f4895041b0924276aa587adbb8dc9e98 P 8feebcdef3ba428ba32e62016f3ed365 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:48.440650 29158 leader_election.cc:288] T f4895041b0924276aa587adbb8dc9e98 P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 4 election: Requested vote from peers 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194:39263), 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195:37025)
I20240501 20:00:48.441305 29233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4895041b0924276aa587adbb8dc9e98" candidate_uuid: "8feebcdef3ba428ba32e62016f3ed365" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "78bf5e3ff675477cb9f9c874292fa63f"
I20240501 20:00:48.441674 29305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4895041b0924276aa587adbb8dc9e98" candidate_uuid: "8feebcdef3ba428ba32e62016f3ed365" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "16f5b5441d8c4a44a9c0a6cca8356822"
I20240501 20:00:48.441781 29233 raft_consensus.cc:3037] T f4895041b0924276aa587adbb8dc9e98 P 78bf5e3ff675477cb9f9c874292fa63f [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:00:48.442106 29305 raft_consensus.cc:3032] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:00:48.442330 29305 raft_consensus.cc:729] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 16f5b5441d8c4a44a9c0a6cca8356822, State: Running, Role: LEADER
I20240501 20:00:48.442869 29305 consensus_queue.cc:260] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:48.443655 29305 raft_consensus.cc:3037] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:00:48.446117 29233 raft_consensus.cc:2454] T f4895041b0924276aa587adbb8dc9e98 P 78bf5e3ff675477cb9f9c874292fa63f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8feebcdef3ba428ba32e62016f3ed365 in term 4.
I20240501 20:00:48.446926 29124 leader_election.cc:302] T f4895041b0924276aa587adbb8dc9e98 P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78bf5e3ff675477cb9f9c874292fa63f, 8feebcdef3ba428ba32e62016f3ed365; no voters:
I20240501 20:00:48.447599 29468 raft_consensus.cc:2781] T f4895041b0924276aa587adbb8dc9e98 P 8feebcdef3ba428ba32e62016f3ed365 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:00:48.447973 29468 raft_consensus.cc:686] T f4895041b0924276aa587adbb8dc9e98 P 8feebcdef3ba428ba32e62016f3ed365 [term 4 LEADER]: Becoming Leader. State: Replica: 8feebcdef3ba428ba32e62016f3ed365, State: Running, Role: LEADER
I20240501 20:00:48.448308 29305 raft_consensus.cc:2454] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8feebcdef3ba428ba32e62016f3ed365 in term 4.
I20240501 20:00:48.448772 29468 consensus_queue.cc:237] T f4895041b0924276aa587adbb8dc9e98 P 8feebcdef3ba428ba32e62016f3ed365 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:48.455391 29055 catalog_manager.cc:5484] T f4895041b0924276aa587adbb8dc9e98 P 8feebcdef3ba428ba32e62016f3ed365 reported cstate change: term changed from 3 to 4, leader changed from 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195) to 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193). New cstate: current_term: 4 leader_uuid: "8feebcdef3ba428ba32e62016f3ed365" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } health_report { overall_health: UNKNOWN } } }
I20240501 20:00:48.555895 29473 raft_consensus.cc:979] T d057ab88d0084efe8d2fe408a42a24b7 P 78bf5e3ff675477cb9f9c874292fa63f: : Instructing follower 8feebcdef3ba428ba32e62016f3ed365 to start an election
I20240501 20:00:48.556340 29482 raft_consensus.cc:1067] T d057ab88d0084efe8d2fe408a42a24b7 P 78bf5e3ff675477cb9f9c874292fa63f [term 2 LEADER]: Signalling peer 8feebcdef3ba428ba32e62016f3ed365 to start an election
I20240501 20:00:48.557574 29158 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d057ab88d0084efe8d2fe408a42a24b7"
dest_uuid: "8feebcdef3ba428ba32e62016f3ed365"
from {username='slave'} at 127.0.0.1:51380
I20240501 20:00:48.557950 29158 raft_consensus.cc:482] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:00:48.558176 29158 raft_consensus.cc:3037] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:00:48.561985 29158 raft_consensus.cc:504] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:48.563266 29158 leader_election.cc:288] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 3 election: Requested vote from peers 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194:39263), 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195:37025)
I20240501 20:00:48.563851 29233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d057ab88d0084efe8d2fe408a42a24b7" candidate_uuid: "8feebcdef3ba428ba32e62016f3ed365" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "78bf5e3ff675477cb9f9c874292fa63f"
I20240501 20:00:48.564142 29305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d057ab88d0084efe8d2fe408a42a24b7" candidate_uuid: "8feebcdef3ba428ba32e62016f3ed365" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "16f5b5441d8c4a44a9c0a6cca8356822"
I20240501 20:00:48.564282 29233 raft_consensus.cc:3032] T d057ab88d0084efe8d2fe408a42a24b7 P 78bf5e3ff675477cb9f9c874292fa63f [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:00:48.564551 29233 raft_consensus.cc:729] T d057ab88d0084efe8d2fe408a42a24b7 P 78bf5e3ff675477cb9f9c874292fa63f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 78bf5e3ff675477cb9f9c874292fa63f, State: Running, Role: LEADER
I20240501 20:00:48.564553 29305 raft_consensus.cc:3037] T d057ab88d0084efe8d2fe408a42a24b7 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:00:48.565124 29233 consensus_queue.cc:260] T d057ab88d0084efe8d2fe408a42a24b7 P 78bf5e3ff675477cb9f9c874292fa63f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:48.565918 29233 raft_consensus.cc:3037] T d057ab88d0084efe8d2fe408a42a24b7 P 78bf5e3ff675477cb9f9c874292fa63f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:00:48.568171 29305 raft_consensus.cc:2454] T d057ab88d0084efe8d2fe408a42a24b7 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8feebcdef3ba428ba32e62016f3ed365 in term 3.
I20240501 20:00:48.568969 29122 leader_election.cc:302] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16f5b5441d8c4a44a9c0a6cca8356822, 8feebcdef3ba428ba32e62016f3ed365; no voters:
I20240501 20:00:48.569517 29468 raft_consensus.cc:2781] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:00:48.569794 29233 raft_consensus.cc:2454] T d057ab88d0084efe8d2fe408a42a24b7 P 78bf5e3ff675477cb9f9c874292fa63f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8feebcdef3ba428ba32e62016f3ed365 in term 3.
I20240501 20:00:48.569904 29468 raft_consensus.cc:686] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365 [term 3 LEADER]: Becoming Leader. State: Replica: 8feebcdef3ba428ba32e62016f3ed365, State: Running, Role: LEADER
I20240501 20:00:48.571060 29468 consensus_queue.cc:237] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:48.578081 29055 catalog_manager.cc:5484] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365 reported cstate change: term changed from 2 to 3, leader changed from 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194) to 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193). New cstate: current_term: 3 leader_uuid: "8feebcdef3ba428ba32e62016f3ed365" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } health_report { overall_health: UNKNOWN } } }
I20240501 20:00:48.707276 29473 raft_consensus.cc:979] T 419e173751fc4cd18b50341160a84d24 P 78bf5e3ff675477cb9f9c874292fa63f: : Instructing follower 8feebcdef3ba428ba32e62016f3ed365 to start an election
I20240501 20:00:48.707692 29482 raft_consensus.cc:1067] T 419e173751fc4cd18b50341160a84d24 P 78bf5e3ff675477cb9f9c874292fa63f [term 2 LEADER]: Signalling peer 8feebcdef3ba428ba32e62016f3ed365 to start an election
I20240501 20:00:48.708814 29158 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "419e173751fc4cd18b50341160a84d24"
dest_uuid: "8feebcdef3ba428ba32e62016f3ed365"
from {username='slave'} at 127.0.0.1:51380
I20240501 20:00:48.709183 29158 raft_consensus.cc:482] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:00:48.709424 29158 raft_consensus.cc:3037] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:00:48.713133 29158 raft_consensus.cc:504] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:48.714219 29158 leader_election.cc:288] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 3 election: Requested vote from peers 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194:39263), 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195:37025)
I20240501 20:00:48.714906 29233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "419e173751fc4cd18b50341160a84d24" candidate_uuid: "8feebcdef3ba428ba32e62016f3ed365" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "78bf5e3ff675477cb9f9c874292fa63f"
I20240501 20:00:48.715102 29305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "419e173751fc4cd18b50341160a84d24" candidate_uuid: "8feebcdef3ba428ba32e62016f3ed365" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "16f5b5441d8c4a44a9c0a6cca8356822"
I20240501 20:00:48.715385 29233 raft_consensus.cc:3032] T 419e173751fc4cd18b50341160a84d24 P 78bf5e3ff675477cb9f9c874292fa63f [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:00:48.715559 29305 raft_consensus.cc:3037] T 419e173751fc4cd18b50341160a84d24 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:00:48.715760 29233 raft_consensus.cc:729] T 419e173751fc4cd18b50341160a84d24 P 78bf5e3ff675477cb9f9c874292fa63f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 78bf5e3ff675477cb9f9c874292fa63f, State: Running, Role: LEADER
I20240501 20:00:48.716419 29233 consensus_queue.cc:260] T 419e173751fc4cd18b50341160a84d24 P 78bf5e3ff675477cb9f9c874292fa63f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:48.717185 29233 raft_consensus.cc:3037] T 419e173751fc4cd18b50341160a84d24 P 78bf5e3ff675477cb9f9c874292fa63f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:00:48.719338 29305 raft_consensus.cc:2454] T 419e173751fc4cd18b50341160a84d24 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8feebcdef3ba428ba32e62016f3ed365 in term 3.
I20240501 20:00:48.720124 29122 leader_election.cc:302] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16f5b5441d8c4a44a9c0a6cca8356822, 8feebcdef3ba428ba32e62016f3ed365; no voters:
I20240501 20:00:48.720806 29468 raft_consensus.cc:2781] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:00:48.721153 29468 raft_consensus.cc:686] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 [term 3 LEADER]: Becoming Leader. State: Replica: 8feebcdef3ba428ba32e62016f3ed365, State: Running, Role: LEADER
I20240501 20:00:48.721199 29233 raft_consensus.cc:2454] T 419e173751fc4cd18b50341160a84d24 P 78bf5e3ff675477cb9f9c874292fa63f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8feebcdef3ba428ba32e62016f3ed365 in term 3.
I20240501 20:00:48.722046 29468 consensus_queue.cc:237] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:48.727893 29055 catalog_manager.cc:5484] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 reported cstate change: term changed from 2 to 3, leader changed from 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194) to 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193). New cstate: current_term: 3 leader_uuid: "8feebcdef3ba428ba32e62016f3ed365" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } health_report { overall_health: UNKNOWN } } }
I20240501 20:00:48.781548 29473 raft_consensus.cc:979] T 27c72ce2560f4d3baf61f0441f5c4129 P 78bf5e3ff675477cb9f9c874292fa63f: : Instructing follower 8feebcdef3ba428ba32e62016f3ed365 to start an election
I20240501 20:00:48.781939 29473 raft_consensus.cc:1067] T 27c72ce2560f4d3baf61f0441f5c4129 P 78bf5e3ff675477cb9f9c874292fa63f [term 2 LEADER]: Signalling peer 8feebcdef3ba428ba32e62016f3ed365 to start an election
I20240501 20:00:48.782969 29158 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "27c72ce2560f4d3baf61f0441f5c4129"
dest_uuid: "8feebcdef3ba428ba32e62016f3ed365"
from {username='slave'} at 127.0.0.1:51380
I20240501 20:00:48.783308 29158 raft_consensus.cc:482] T 27c72ce2560f4d3baf61f0441f5c4129 P 8feebcdef3ba428ba32e62016f3ed365 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:00:48.783512 29158 raft_consensus.cc:3037] T 27c72ce2560f4d3baf61f0441f5c4129 P 8feebcdef3ba428ba32e62016f3ed365 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:00:48.787014 29158 raft_consensus.cc:504] T 27c72ce2560f4d3baf61f0441f5c4129 P 8feebcdef3ba428ba32e62016f3ed365 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:48.788172 29158 leader_election.cc:288] T 27c72ce2560f4d3baf61f0441f5c4129 P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 3 election: Requested vote from peers 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194:39263), 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195:37025)
I20240501 20:00:48.788767 29233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c72ce2560f4d3baf61f0441f5c4129" candidate_uuid: "8feebcdef3ba428ba32e62016f3ed365" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "78bf5e3ff675477cb9f9c874292fa63f"
I20240501 20:00:48.788916 29305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27c72ce2560f4d3baf61f0441f5c4129" candidate_uuid: "8feebcdef3ba428ba32e62016f3ed365" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "16f5b5441d8c4a44a9c0a6cca8356822"
I20240501 20:00:48.789294 29233 raft_consensus.cc:3032] T 27c72ce2560f4d3baf61f0441f5c4129 P 78bf5e3ff675477cb9f9c874292fa63f [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:00:48.789351 29305 raft_consensus.cc:3037] T 27c72ce2560f4d3baf61f0441f5c4129 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:00:48.789683 29233 raft_consensus.cc:729] T 27c72ce2560f4d3baf61f0441f5c4129 P 78bf5e3ff675477cb9f9c874292fa63f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 78bf5e3ff675477cb9f9c874292fa63f, State: Running, Role: LEADER
I20240501 20:00:48.790410 29233 consensus_queue.cc:260] T 27c72ce2560f4d3baf61f0441f5c4129 P 78bf5e3ff675477cb9f9c874292fa63f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:48.791473 29233 raft_consensus.cc:3037] T 27c72ce2560f4d3baf61f0441f5c4129 P 78bf5e3ff675477cb9f9c874292fa63f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:00:48.794620 29305 raft_consensus.cc:2454] T 27c72ce2560f4d3baf61f0441f5c4129 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8feebcdef3ba428ba32e62016f3ed365 in term 3.
I20240501 20:00:48.795408 29122 leader_election.cc:302] T 27c72ce2560f4d3baf61f0441f5c4129 P 8feebcdef3ba428ba32e62016f3ed365 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16f5b5441d8c4a44a9c0a6cca8356822, 8feebcdef3ba428ba32e62016f3ed365; no voters:
I20240501 20:00:48.795933 29468 raft_consensus.cc:2781] T 27c72ce2560f4d3baf61f0441f5c4129 P 8feebcdef3ba428ba32e62016f3ed365 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:00:48.796406 29468 raft_consensus.cc:686] T 27c72ce2560f4d3baf61f0441f5c4129 P 8feebcdef3ba428ba32e62016f3ed365 [term 3 LEADER]: Becoming Leader. State: Replica: 8feebcdef3ba428ba32e62016f3ed365, State: Running, Role: LEADER
I20240501 20:00:48.796387 29233 raft_consensus.cc:2454] T 27c72ce2560f4d3baf61f0441f5c4129 P 78bf5e3ff675477cb9f9c874292fa63f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8feebcdef3ba428ba32e62016f3ed365 in term 3.
I20240501 20:00:48.797230 29468 consensus_queue.cc:237] T 27c72ce2560f4d3baf61f0441f5c4129 P 8feebcdef3ba428ba32e62016f3ed365 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:48.803254 29055 catalog_manager.cc:5484] T 27c72ce2560f4d3baf61f0441f5c4129 P 8feebcdef3ba428ba32e62016f3ed365 reported cstate change: term changed from 2 to 3, leader changed from 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194) to 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193). New cstate: current_term: 3 leader_uuid: "8feebcdef3ba428ba32e62016f3ed365" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } health_report { overall_health: UNKNOWN } } }
I20240501 20:00:48.868162 29233 raft_consensus.cc:1261] T 3d51e8d55df849e5ac6e5acdc42ac63c P 78bf5e3ff675477cb9f9c874292fa63f [term 4 FOLLOWER]: Refusing update from remote peer 8feebcdef3ba428ba32e62016f3ed365: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240501 20:00:48.869091 29468 consensus_queue.cc:1035] T 3d51e8d55df849e5ac6e5acdc42ac63c P 8feebcdef3ba428ba32e62016f3ed365 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:00:48.877580 29305 raft_consensus.cc:1261] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 [term 4 FOLLOWER]: Refusing update from remote peer 8feebcdef3ba428ba32e62016f3ed365: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240501 20:00:48.879655 29488 consensus_queue.cc:1035] T 3d51e8d55df849e5ac6e5acdc42ac63c P 8feebcdef3ba428ba32e62016f3ed365 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240501 20:00:48.964978 29305 raft_consensus.cc:1261] T d057ab88d0084efe8d2fe408a42a24b7 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 3 FOLLOWER]: Refusing update from remote peer 8feebcdef3ba428ba32e62016f3ed365: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 20:00:48.965965 29488 consensus_queue.cc:1035] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:00:48.975386 29233 raft_consensus.cc:1261] T d057ab88d0084efe8d2fe408a42a24b7 P 78bf5e3ff675477cb9f9c874292fa63f [term 3 FOLLOWER]: Refusing update from remote peer 8feebcdef3ba428ba32e62016f3ed365: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 20:00:48.976501 29488 consensus_queue.cc:1035] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:00:49.057282 29233 raft_consensus.cc:1261] T f4895041b0924276aa587adbb8dc9e98 P 78bf5e3ff675477cb9f9c874292fa63f [term 4 FOLLOWER]: Refusing update from remote peer 8feebcdef3ba428ba32e62016f3ed365: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240501 20:00:49.058215 29488 consensus_queue.cc:1035] T f4895041b0924276aa587adbb8dc9e98 P 8feebcdef3ba428ba32e62016f3ed365 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:00:49.067221 29305 raft_consensus.cc:1261] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 4 FOLLOWER]: Refusing update from remote peer 8feebcdef3ba428ba32e62016f3ed365: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240501 20:00:49.068325 29488 consensus_queue.cc:1035] T f4895041b0924276aa587adbb8dc9e98 P 8feebcdef3ba428ba32e62016f3ed365 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:00:49.101455 29233 raft_consensus.cc:1261] T 419e173751fc4cd18b50341160a84d24 P 78bf5e3ff675477cb9f9c874292fa63f [term 3 FOLLOWER]: Refusing update from remote peer 8feebcdef3ba428ba32e62016f3ed365: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 20:00:49.102373 29488 consensus_queue.cc:1035] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:00:49.109496 29305 raft_consensus.cc:1261] T 419e173751fc4cd18b50341160a84d24 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 3 FOLLOWER]: Refusing update from remote peer 8feebcdef3ba428ba32e62016f3ed365: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 20:00:49.110939 29495 consensus_queue.cc:1035] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:00:49.288771 29233 raft_consensus.cc:1261] T 27c72ce2560f4d3baf61f0441f5c4129 P 78bf5e3ff675477cb9f9c874292fa63f [term 3 FOLLOWER]: Refusing update from remote peer 8feebcdef3ba428ba32e62016f3ed365: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 20:00:49.289746 29495 consensus_queue.cc:1035] T 27c72ce2560f4d3baf61f0441f5c4129 P 8feebcdef3ba428ba32e62016f3ed365 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:00:49.298017 29305 raft_consensus.cc:1261] T 27c72ce2560f4d3baf61f0441f5c4129 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 3 FOLLOWER]: Refusing update from remote peer 8feebcdef3ba428ba32e62016f3ed365: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 20:00:49.299058 29475 consensus_queue.cc:1035] T 27c72ce2560f4d3baf61f0441f5c4129 P 8feebcdef3ba428ba32e62016f3ed365 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:00:51.329679 28519 auto_leader_rebalancer-test.cc:365] The leader distribution is
I20240501 20:00:51.330230 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:51.337121 29158 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0d9b61e9c98e4d19b9272fe70fab0c3d"
dest_uuid: "8feebcdef3ba428ba32e62016f3ed365"
mode: GRACEFUL
new_leader_uuid: "16f5b5441d8c4a44a9c0a6cca8356822"
from {username='slave'} at 127.0.0.1:48854
I20240501 20:00:51.337576 29158 raft_consensus.cc:595] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 8feebcdef3ba428ba32e62016f3ed365 [term 2 LEADER]: Received request to transfer leadership to TS 16f5b5441d8c4a44a9c0a6cca8356822
I20240501 20:00:51.339604 29158 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "419e173751fc4cd18b50341160a84d24"
dest_uuid: "8feebcdef3ba428ba32e62016f3ed365"
mode: GRACEFUL
new_leader_uuid: "16f5b5441d8c4a44a9c0a6cca8356822"
from {username='slave'} at 127.0.0.1:48854
I20240501 20:00:51.340008 29158 raft_consensus.cc:595] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 [term 3 LEADER]: Received request to transfer leadership to TS 16f5b5441d8c4a44a9c0a6cca8356822
I20240501 20:00:51.341887 29158 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5bd59b3e54144bd2986538a355cadd9b"
dest_uuid: "8feebcdef3ba428ba32e62016f3ed365"
mode: GRACEFUL
new_leader_uuid: "78bf5e3ff675477cb9f9c874292fa63f"
from {username='slave'} at 127.0.0.1:48854
I20240501 20:00:51.342249 29158 raft_consensus.cc:595] T 5bd59b3e54144bd2986538a355cadd9b P 8feebcdef3ba428ba32e62016f3ed365 [term 2 LEADER]: Received request to transfer leadership to TS 78bf5e3ff675477cb9f9c874292fa63f
I20240501 20:00:51.344458 29158 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "afbcfea6a7894b5ab7c122d3d4134869"
dest_uuid: "8feebcdef3ba428ba32e62016f3ed365"
mode: GRACEFUL
new_leader_uuid: "78bf5e3ff675477cb9f9c874292fa63f"
from {username='slave'} at 127.0.0.1:48854
I20240501 20:00:51.344837 29158 raft_consensus.cc:595] T afbcfea6a7894b5ab7c122d3d4134869 P 8feebcdef3ba428ba32e62016f3ed365 [term 2 LEADER]: Received request to transfer leadership to TS 78bf5e3ff675477cb9f9c874292fa63f
I20240501 20:00:51.345656 28519 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 4
I20240501 20:00:51.346192 28519 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:51.368710 29501 raft_consensus.cc:979] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365: : Instructing follower 16f5b5441d8c4a44a9c0a6cca8356822 to start an election
I20240501 20:00:51.369141 29495 raft_consensus.cc:1067] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 [term 3 LEADER]: Signalling peer 16f5b5441d8c4a44a9c0a6cca8356822 to start an election
I20240501 20:00:51.370318 29305 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "419e173751fc4cd18b50341160a84d24"
dest_uuid: "16f5b5441d8c4a44a9c0a6cca8356822"
from {username='slave'} at 127.0.0.1:39788
I20240501 20:00:51.370707 29305 raft_consensus.cc:482] T 419e173751fc4cd18b50341160a84d24 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:00:51.370944 29305 raft_consensus.cc:3037] T 419e173751fc4cd18b50341160a84d24 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:00:51.374861 29305 raft_consensus.cc:504] T 419e173751fc4cd18b50341160a84d24 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:51.376600 29305 leader_election.cc:288] T 419e173751fc4cd18b50341160a84d24 P 16f5b5441d8c4a44a9c0a6cca8356822 [CANDIDATE]: Term 4 election: Requested vote from peers 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193:43937), 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194:39263)
I20240501 20:00:51.377200 29158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "419e173751fc4cd18b50341160a84d24" candidate_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "8feebcdef3ba428ba32e62016f3ed365"
I20240501 20:00:51.377446 29233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "419e173751fc4cd18b50341160a84d24" candidate_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "78bf5e3ff675477cb9f9c874292fa63f"
I20240501 20:00:51.377748 29158 raft_consensus.cc:3032] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:00:51.377908 29233 raft_consensus.cc:3037] T 419e173751fc4cd18b50341160a84d24 P 78bf5e3ff675477cb9f9c874292fa63f [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:00:51.378055 29158 raft_consensus.cc:729] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8feebcdef3ba428ba32e62016f3ed365, State: Running, Role: LEADER
I20240501 20:00:51.378774 29158 consensus_queue.cc:260] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:51.379858 29158 raft_consensus.cc:3037] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:00:51.383512 29233 raft_consensus.cc:2454] T 419e173751fc4cd18b50341160a84d24 P 78bf5e3ff675477cb9f9c874292fa63f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16f5b5441d8c4a44a9c0a6cca8356822 in term 4.
I20240501 20:00:51.384523 29273 leader_election.cc:302] T 419e173751fc4cd18b50341160a84d24 P 16f5b5441d8c4a44a9c0a6cca8356822 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16f5b5441d8c4a44a9c0a6cca8356822, 78bf5e3ff675477cb9f9c874292fa63f; no voters:
I20240501 20:00:51.385169 29471 raft_consensus.cc:2781] T 419e173751fc4cd18b50341160a84d24 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:00:51.385372 29158 raft_consensus.cc:2454] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16f5b5441d8c4a44a9c0a6cca8356822 in term 4.
I20240501 20:00:51.386126 29471 raft_consensus.cc:686] T 419e173751fc4cd18b50341160a84d24 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 4 LEADER]: Becoming Leader. State: Replica: 16f5b5441d8c4a44a9c0a6cca8356822, State: Running, Role: LEADER
I20240501 20:00:51.386824 29471 consensus_queue.cc:237] T 419e173751fc4cd18b50341160a84d24 P 16f5b5441d8c4a44a9c0a6cca8356822 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:51.392220 29054 catalog_manager.cc:5484] T 419e173751fc4cd18b50341160a84d24 P 16f5b5441d8c4a44a9c0a6cca8356822 reported cstate change: term changed from 3 to 4, leader changed from 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193) to 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195). New cstate: current_term: 4 leader_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } health_report { overall_health: HEALTHY } } }
I20240501 20:00:51.501541 29501 raft_consensus.cc:979] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 8feebcdef3ba428ba32e62016f3ed365: : Instructing follower 16f5b5441d8c4a44a9c0a6cca8356822 to start an election
I20240501 20:00:51.502002 29495 raft_consensus.cc:1067] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 8feebcdef3ba428ba32e62016f3ed365 [term 2 LEADER]: Signalling peer 16f5b5441d8c4a44a9c0a6cca8356822 to start an election
I20240501 20:00:51.503198 29305 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0d9b61e9c98e4d19b9272fe70fab0c3d"
dest_uuid: "16f5b5441d8c4a44a9c0a6cca8356822"
from {username='slave'} at 127.0.0.1:39788
I20240501 20:00:51.503559 29305 raft_consensus.cc:482] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 16f5b5441d8c4a44a9c0a6cca8356822 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:00:51.503755 29305 raft_consensus.cc:3037] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 16f5b5441d8c4a44a9c0a6cca8356822 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:00:51.507395 29305 raft_consensus.cc:504] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 16f5b5441d8c4a44a9c0a6cca8356822 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } } peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } }
I20240501 20:00:51.508498 29305 leader_election.cc:288] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 16f5b5441d8c4a44a9c0a6cca8356822 [CANDIDATE]: Term 3 election: Requested vote from peers 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193:43937), 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194:39263)
I20240501 20:00:51.509150 29158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d9b61e9c98e4d19b9272fe70fab0c3d" candidate_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8feebcdef3ba428ba32e62016f3ed365"
I20240501 20:00:51.509346 29233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d9b61e9c98e4d19b9272fe70fab0c3d" candidate_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "78bf5e3ff675477cb9f9c874292fa63f"
I20240501 20:00:51.509691 29158 raft_consensus.cc:3032] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 8feebcdef3ba428ba32e62016f3ed365 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:00:51.509809 29233 raft_consensus.cc:3037] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 78bf5e3ff675477cb9f9c874292fa63f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:00:51.510008 29158 raft_consensus.cc:729] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 8feebcdef3ba428ba32e62016f3ed365 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8feebcdef3ba428ba32e62016f3ed365, State: Running, Role: LEADER
I20240501 20:00:51.510751 29158 consensus_queue.cc:260] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 8feebcdef3ba428ba32e62016f3ed365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } } peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } }
I20240501 20:00:51.511837 29158 raft_consensus.cc:3037] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 8feebcdef3ba428ba32e62016f3ed365 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:00:51.515074 29233 raft_consensus.cc:2454] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 78bf5e3ff675477cb9f9c874292fa63f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16f5b5441d8c4a44a9c0a6cca8356822 in term 3.
I20240501 20:00:51.515973 29273 leader_election.cc:302] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 16f5b5441d8c4a44a9c0a6cca8356822 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16f5b5441d8c4a44a9c0a6cca8356822, 78bf5e3ff675477cb9f9c874292fa63f; no voters:
I20240501 20:00:51.516644 29471 raft_consensus.cc:2781] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 16f5b5441d8c4a44a9c0a6cca8356822 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:00:51.516996 29471 raft_consensus.cc:686] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 16f5b5441d8c4a44a9c0a6cca8356822 [term 3 LEADER]: Becoming Leader. State: Replica: 16f5b5441d8c4a44a9c0a6cca8356822, State: Running, Role: LEADER
I20240501 20:00:51.517071 29158 raft_consensus.cc:2454] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 8feebcdef3ba428ba32e62016f3ed365 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16f5b5441d8c4a44a9c0a6cca8356822 in term 3.
I20240501 20:00:51.517849 29471 consensus_queue.cc:237] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 16f5b5441d8c4a44a9c0a6cca8356822 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } } peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } }
I20240501 20:00:51.524500 29054 catalog_manager.cc:5484] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 16f5b5441d8c4a44a9c0a6cca8356822 reported cstate change: term changed from 2 to 3, leader changed from 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193) to 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195). New cstate: current_term: 3 leader_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } health_report { overall_health: UNKNOWN } } }
I20240501 20:00:51.804787 29501 raft_consensus.cc:979] T afbcfea6a7894b5ab7c122d3d4134869 P 8feebcdef3ba428ba32e62016f3ed365: : Instructing follower 78bf5e3ff675477cb9f9c874292fa63f to start an election
I20240501 20:00:51.805168 29495 raft_consensus.cc:1067] T afbcfea6a7894b5ab7c122d3d4134869 P 8feebcdef3ba428ba32e62016f3ed365 [term 2 LEADER]: Signalling peer 78bf5e3ff675477cb9f9c874292fa63f to start an election
I20240501 20:00:51.806205 29233 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "afbcfea6a7894b5ab7c122d3d4134869"
dest_uuid: "78bf5e3ff675477cb9f9c874292fa63f"
from {username='slave'} at 127.0.0.1:42828
I20240501 20:00:51.806573 29233 raft_consensus.cc:482] T afbcfea6a7894b5ab7c122d3d4134869 P 78bf5e3ff675477cb9f9c874292fa63f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:00:51.806794 29233 raft_consensus.cc:3037] T afbcfea6a7894b5ab7c122d3d4134869 P 78bf5e3ff675477cb9f9c874292fa63f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:00:51.810384 29233 raft_consensus.cc:504] T afbcfea6a7894b5ab7c122d3d4134869 P 78bf5e3ff675477cb9f9c874292fa63f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:51.811857 29233 leader_election.cc:288] T afbcfea6a7894b5ab7c122d3d4134869 P 78bf5e3ff675477cb9f9c874292fa63f [CANDIDATE]: Term 3 election: Requested vote from peers 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193:43937), 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195:37025)
I20240501 20:00:51.812572 29158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afbcfea6a7894b5ab7c122d3d4134869" candidate_uuid: "78bf5e3ff675477cb9f9c874292fa63f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8feebcdef3ba428ba32e62016f3ed365"
I20240501 20:00:51.812664 29305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afbcfea6a7894b5ab7c122d3d4134869" candidate_uuid: "78bf5e3ff675477cb9f9c874292fa63f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "16f5b5441d8c4a44a9c0a6cca8356822"
I20240501 20:00:51.813105 29158 raft_consensus.cc:3032] T afbcfea6a7894b5ab7c122d3d4134869 P 8feebcdef3ba428ba32e62016f3ed365 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:00:51.813143 29305 raft_consensus.cc:3037] T afbcfea6a7894b5ab7c122d3d4134869 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:00:51.813479 29158 raft_consensus.cc:729] T afbcfea6a7894b5ab7c122d3d4134869 P 8feebcdef3ba428ba32e62016f3ed365 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8feebcdef3ba428ba32e62016f3ed365, State: Running, Role: LEADER
I20240501 20:00:51.814134 29158 consensus_queue.cc:260] T afbcfea6a7894b5ab7c122d3d4134869 P 8feebcdef3ba428ba32e62016f3ed365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:51.815202 29158 raft_consensus.cc:3037] T afbcfea6a7894b5ab7c122d3d4134869 P 8feebcdef3ba428ba32e62016f3ed365 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:00:51.818684 29305 raft_consensus.cc:2454] T afbcfea6a7894b5ab7c122d3d4134869 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78bf5e3ff675477cb9f9c874292fa63f in term 3.
I20240501 20:00:51.819684 29197 leader_election.cc:302] T afbcfea6a7894b5ab7c122d3d4134869 P 78bf5e3ff675477cb9f9c874292fa63f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16f5b5441d8c4a44a9c0a6cca8356822, 78bf5e3ff675477cb9f9c874292fa63f; no voters:
I20240501 20:00:51.820291 29473 raft_consensus.cc:2781] T afbcfea6a7894b5ab7c122d3d4134869 P 78bf5e3ff675477cb9f9c874292fa63f [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:00:51.820346 29158 raft_consensus.cc:2454] T afbcfea6a7894b5ab7c122d3d4134869 P 8feebcdef3ba428ba32e62016f3ed365 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78bf5e3ff675477cb9f9c874292fa63f in term 3.
I20240501 20:00:51.821580 29473 raft_consensus.cc:686] T afbcfea6a7894b5ab7c122d3d4134869 P 78bf5e3ff675477cb9f9c874292fa63f [term 3 LEADER]: Becoming Leader. State: Replica: 78bf5e3ff675477cb9f9c874292fa63f, State: Running, Role: LEADER
I20240501 20:00:51.822607 29473 consensus_queue.cc:237] T afbcfea6a7894b5ab7c122d3d4134869 P 78bf5e3ff675477cb9f9c874292fa63f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:51.826428 29233 raft_consensus.cc:1261] T 419e173751fc4cd18b50341160a84d24 P 78bf5e3ff675477cb9f9c874292fa63f [term 4 FOLLOWER]: Refusing update from remote peer 16f5b5441d8c4a44a9c0a6cca8356822: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240501 20:00:51.828063 29471 consensus_queue.cc:1035] T 419e173751fc4cd18b50341160a84d24 P 16f5b5441d8c4a44a9c0a6cca8356822 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:00:51.829528 29054 catalog_manager.cc:5484] T afbcfea6a7894b5ab7c122d3d4134869 P 78bf5e3ff675477cb9f9c874292fa63f reported cstate change: term changed from 2 to 3, leader changed from 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193) to 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194). New cstate: current_term: 3 leader_uuid: "78bf5e3ff675477cb9f9c874292fa63f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } health_report { overall_health: UNKNOWN } } }
I20240501 20:00:51.835755 29158 raft_consensus.cc:1261] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 [term 4 FOLLOWER]: Refusing update from remote peer 16f5b5441d8c4a44a9c0a6cca8356822: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240501 20:00:51.837309 29471 consensus_queue.cc:1035] T 419e173751fc4cd18b50341160a84d24 P 16f5b5441d8c4a44a9c0a6cca8356822 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:00:51.880934 29501 raft_consensus.cc:979] T 5bd59b3e54144bd2986538a355cadd9b P 8feebcdef3ba428ba32e62016f3ed365: : Instructing follower 78bf5e3ff675477cb9f9c874292fa63f to start an election
I20240501 20:00:51.881354 29495 raft_consensus.cc:1067] T 5bd59b3e54144bd2986538a355cadd9b P 8feebcdef3ba428ba32e62016f3ed365 [term 2 LEADER]: Signalling peer 78bf5e3ff675477cb9f9c874292fa63f to start an election
I20240501 20:00:51.882436 29233 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5bd59b3e54144bd2986538a355cadd9b"
dest_uuid: "78bf5e3ff675477cb9f9c874292fa63f"
from {username='slave'} at 127.0.0.1:42828
I20240501 20:00:51.882791 29233 raft_consensus.cc:482] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:00:51.883008 29233 raft_consensus.cc:3037] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:00:51.887112 29233 raft_consensus.cc:504] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:51.888262 29233 leader_election.cc:288] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f [CANDIDATE]: Term 3 election: Requested vote from peers 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193:43937), 16f5b5441d8c4a44a9c0a6cca8356822 (127.27.217.195:37025)
I20240501 20:00:51.889078 29305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bd59b3e54144bd2986538a355cadd9b" candidate_uuid: "78bf5e3ff675477cb9f9c874292fa63f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "16f5b5441d8c4a44a9c0a6cca8356822"
I20240501 20:00:51.889109 29158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bd59b3e54144bd2986538a355cadd9b" candidate_uuid: "78bf5e3ff675477cb9f9c874292fa63f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8feebcdef3ba428ba32e62016f3ed365"
I20240501 20:00:51.889534 29305 raft_consensus.cc:3037] T 5bd59b3e54144bd2986538a355cadd9b P 16f5b5441d8c4a44a9c0a6cca8356822 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:00:51.889597 29158 raft_consensus.cc:3032] T 5bd59b3e54144bd2986538a355cadd9b P 8feebcdef3ba428ba32e62016f3ed365 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:00:51.889874 29158 raft_consensus.cc:729] T 5bd59b3e54144bd2986538a355cadd9b P 8feebcdef3ba428ba32e62016f3ed365 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8feebcdef3ba428ba32e62016f3ed365, State: Running, Role: LEADER
I20240501 20:00:51.890362 29158 consensus_queue.cc:260] T 5bd59b3e54144bd2986538a355cadd9b P 8feebcdef3ba428ba32e62016f3ed365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:51.891147 29158 raft_consensus.cc:3037] T 5bd59b3e54144bd2986538a355cadd9b P 8feebcdef3ba428ba32e62016f3ed365 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:00:51.893141 29305 raft_consensus.cc:2454] T 5bd59b3e54144bd2986538a355cadd9b P 16f5b5441d8c4a44a9c0a6cca8356822 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78bf5e3ff675477cb9f9c874292fa63f in term 3.
I20240501 20:00:51.893823 29197 leader_election.cc:302] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16f5b5441d8c4a44a9c0a6cca8356822, 78bf5e3ff675477cb9f9c874292fa63f; no voters:
I20240501 20:00:51.894336 29473 raft_consensus.cc:2781] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:00:51.894762 29473 raft_consensus.cc:686] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f [term 3 LEADER]: Becoming Leader. State: Replica: 78bf5e3ff675477cb9f9c874292fa63f, State: Running, Role: LEADER
I20240501 20:00:51.895009 29158 raft_consensus.cc:2454] T 5bd59b3e54144bd2986538a355cadd9b P 8feebcdef3ba428ba32e62016f3ed365 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78bf5e3ff675477cb9f9c874292fa63f in term 3.
I20240501 20:00:51.895599 29473 consensus_queue.cc:237] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } }
I20240501 20:00:51.900987 29054 catalog_manager.cc:5484] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f reported cstate change: term changed from 2 to 3, leader changed from 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193) to 78bf5e3ff675477cb9f9c874292fa63f (127.27.217.194). New cstate: current_term: 3 leader_uuid: "78bf5e3ff675477cb9f9c874292fa63f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 } health_report { overall_health: UNKNOWN } } }
I20240501 20:00:51.915724 29158 raft_consensus.cc:1261] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 8feebcdef3ba428ba32e62016f3ed365 [term 3 FOLLOWER]: Refusing update from remote peer 16f5b5441d8c4a44a9c0a6cca8356822: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 20:00:51.916575 29471 consensus_queue.cc:1035] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 16f5b5441d8c4a44a9c0a6cca8356822 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:00:51.923292 29233 raft_consensus.cc:1261] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 78bf5e3ff675477cb9f9c874292fa63f [term 3 FOLLOWER]: Refusing update from remote peer 16f5b5441d8c4a44a9c0a6cca8356822: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 20:00:51.924993 29518 consensus_queue.cc:1035] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 16f5b5441d8c4a44a9c0a6cca8356822 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78bf5e3ff675477cb9f9c874292fa63f" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 39263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:00:52.217780 29158 raft_consensus.cc:1261] T afbcfea6a7894b5ab7c122d3d4134869 P 8feebcdef3ba428ba32e62016f3ed365 [term 3 FOLLOWER]: Refusing update from remote peer 78bf5e3ff675477cb9f9c874292fa63f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 20:00:52.218670 29473 consensus_queue.cc:1035] T afbcfea6a7894b5ab7c122d3d4134869 P 78bf5e3ff675477cb9f9c874292fa63f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:00:52.226192 29305 raft_consensus.cc:1261] T afbcfea6a7894b5ab7c122d3d4134869 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 3 FOLLOWER]: Refusing update from remote peer 78bf5e3ff675477cb9f9c874292fa63f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 20:00:52.227576 29473 consensus_queue.cc:1035] T afbcfea6a7894b5ab7c122d3d4134869 P 78bf5e3ff675477cb9f9c874292fa63f [LEADER]: Connected to new peer: Peer: permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:00:52.347005 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:52.351847 28519 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:52.352223 28519 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:52.391849 29158 raft_consensus.cc:1261] T 5bd59b3e54144bd2986538a355cadd9b P 8feebcdef3ba428ba32e62016f3ed365 [term 3 FOLLOWER]: Refusing update from remote peer 78bf5e3ff675477cb9f9c874292fa63f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 20:00:52.392875 29473 consensus_queue.cc:1035] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8feebcdef3ba428ba32e62016f3ed365" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:00:52.399720 29305 raft_consensus.cc:1261] T 5bd59b3e54144bd2986538a355cadd9b P 16f5b5441d8c4a44a9c0a6cca8356822 [term 3 FOLLOWER]: Refusing update from remote peer 78bf5e3ff675477cb9f9c874292fa63f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 20:00:52.401238 29509 consensus_queue.cc:1035] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f [LEADER]: Connected to new peer: Peer: permanent_uuid: "16f5b5441d8c4a44a9c0a6cca8356822" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 37025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:00:53.353183 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:53.359066 28519 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:53.359434 28519 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:54.360209 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:54.365257 28519 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:54.365665 28519 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:55.366353 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:55.371508 28519 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:55.371897 28519 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:56.372701 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:56.377704 28519 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:56.378103 28519 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:57.378906 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:57.385087 28519 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:57.385473 28519 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:58.386236 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:58.392645 28519 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:58.393177 28519 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:00:59.393896 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:00:59.398973 28519 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:00:59.399358 28519 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:01:00.400279 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:01:00.405364 28519 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 20:01:00.405755 28519 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:01:01.427995 28519 auto_leader_rebalancer-test.cc:376] The leader distribution is
I20240501 20:01:01.428519 28519 tablet_server.cc:178] TabletServer@127.27.217.193:0 shutting down...
I20240501 20:01:01.459573 28519 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:01:01.460472 28519 tablet_replica.cc:330] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 8feebcdef3ba428ba32e62016f3ed365: stopping tablet replica
I20240501 20:01:01.461211 28519 raft_consensus.cc:2229] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 8feebcdef3ba428ba32e62016f3ed365 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:01.461808 28519 raft_consensus.cc:2258] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 8feebcdef3ba428ba32e62016f3ed365 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:01.463789 28519 tablet_replica.cc:330] T f4895041b0924276aa587adbb8dc9e98 P 8feebcdef3ba428ba32e62016f3ed365: stopping tablet replica
I20240501 20:01:01.464372 28519 raft_consensus.cc:2229] T f4895041b0924276aa587adbb8dc9e98 P 8feebcdef3ba428ba32e62016f3ed365 [term 4 LEADER]: Raft consensus shutting down.
I20240501 20:01:01.465164 28519 raft_consensus.cc:2258] T f4895041b0924276aa587adbb8dc9e98 P 8feebcdef3ba428ba32e62016f3ed365 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:01.466885 28519 tablet_replica.cc:330] T 27c72ce2560f4d3baf61f0441f5c4129 P 8feebcdef3ba428ba32e62016f3ed365: stopping tablet replica
I20240501 20:01:01.467367 28519 raft_consensus.cc:2229] T 27c72ce2560f4d3baf61f0441f5c4129 P 8feebcdef3ba428ba32e62016f3ed365 [term 3 LEADER]: Raft consensus shutting down.
I20240501 20:01:01.468183 28519 raft_consensus.cc:2258] T 27c72ce2560f4d3baf61f0441f5c4129 P 8feebcdef3ba428ba32e62016f3ed365 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:01.469918 28519 tablet_replica.cc:330] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365: stopping tablet replica
I20240501 20:01:01.470489 28519 raft_consensus.cc:2229] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365 [term 3 LEADER]: Raft consensus shutting down.
I20240501 20:01:01.471244 28519 raft_consensus.cc:2258] T d057ab88d0084efe8d2fe408a42a24b7 P 8feebcdef3ba428ba32e62016f3ed365 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:01.472975 28519 tablet_replica.cc:330] T 5bd59b3e54144bd2986538a355cadd9b P 8feebcdef3ba428ba32e62016f3ed365: stopping tablet replica
I20240501 20:01:01.473456 28519 raft_consensus.cc:2229] T 5bd59b3e54144bd2986538a355cadd9b P 8feebcdef3ba428ba32e62016f3ed365 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:01.473857 28519 raft_consensus.cc:2258] T 5bd59b3e54144bd2986538a355cadd9b P 8feebcdef3ba428ba32e62016f3ed365 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:01.475502 28519 tablet_replica.cc:330] T 3d51e8d55df849e5ac6e5acdc42ac63c P 8feebcdef3ba428ba32e62016f3ed365: stopping tablet replica
I20240501 20:01:01.475984 28519 raft_consensus.cc:2229] T 3d51e8d55df849e5ac6e5acdc42ac63c P 8feebcdef3ba428ba32e62016f3ed365 [term 4 LEADER]: Raft consensus shutting down.
I20240501 20:01:01.476827 28519 raft_consensus.cc:2258] T 3d51e8d55df849e5ac6e5acdc42ac63c P 8feebcdef3ba428ba32e62016f3ed365 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:01.478485 28519 tablet_replica.cc:330] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 8feebcdef3ba428ba32e62016f3ed365: stopping tablet replica
I20240501 20:01:01.478926 28519 raft_consensus.cc:2229] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 8feebcdef3ba428ba32e62016f3ed365 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:01.479334 28519 raft_consensus.cc:2258] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 8feebcdef3ba428ba32e62016f3ed365 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:01.481011 28519 tablet_replica.cc:330] T afbcfea6a7894b5ab7c122d3d4134869 P 8feebcdef3ba428ba32e62016f3ed365: stopping tablet replica
I20240501 20:01:01.481451 28519 raft_consensus.cc:2229] T afbcfea6a7894b5ab7c122d3d4134869 P 8feebcdef3ba428ba32e62016f3ed365 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:01.481866 28519 raft_consensus.cc:2258] T afbcfea6a7894b5ab7c122d3d4134869 P 8feebcdef3ba428ba32e62016f3ed365 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:01.483428 28519 tablet_replica.cc:330] T eed1ac06c82147e3bb681f4bcd1d8688 P 8feebcdef3ba428ba32e62016f3ed365: stopping tablet replica
I20240501 20:01:01.483896 28519 raft_consensus.cc:2229] T eed1ac06c82147e3bb681f4bcd1d8688 P 8feebcdef3ba428ba32e62016f3ed365 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:01.484373 28519 raft_consensus.cc:2258] T eed1ac06c82147e3bb681f4bcd1d8688 P 8feebcdef3ba428ba32e62016f3ed365 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:01.485931 28519 tablet_replica.cc:330] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365: stopping tablet replica
I20240501 20:01:01.486361 28519 raft_consensus.cc:2229] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:01.486768 28519 raft_consensus.cc:2258] T 419e173751fc4cd18b50341160a84d24 P 8feebcdef3ba428ba32e62016f3ed365 [term 4 FOLLOWER]: Raft consensus is shut down!
W20240501 20:01:01.491498 29199 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 4 similar messages]
W20240501 20:01:01.494593 29199 consensus_peers.cc:485] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f -> Peer 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193:43937): Couldn't send request to peer 8feebcdef3ba428ba32e62016f3ed365. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:01.518329 28519 tablet_server.cc:194] TabletServer@127.27.217.193:0 shutdown complete.
I20240501 20:01:01.536744 28519 tablet_server.cc:178] TabletServer@127.27.217.194:0 shutting down...
I20240501 20:01:01.563660 28519 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:01:01.564599 28519 tablet_replica.cc:330] T 419e173751fc4cd18b50341160a84d24 P 78bf5e3ff675477cb9f9c874292fa63f: stopping tablet replica
I20240501 20:01:01.565287 28519 raft_consensus.cc:2229] T 419e173751fc4cd18b50341160a84d24 P 78bf5e3ff675477cb9f9c874292fa63f [term 4 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:01.565748 28519 raft_consensus.cc:2258] T 419e173751fc4cd18b50341160a84d24 P 78bf5e3ff675477cb9f9c874292fa63f [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:01.567499 28519 tablet_replica.cc:330] T 27c72ce2560f4d3baf61f0441f5c4129 P 78bf5e3ff675477cb9f9c874292fa63f: stopping tablet replica
I20240501 20:01:01.568027 28519 raft_consensus.cc:2229] T 27c72ce2560f4d3baf61f0441f5c4129 P 78bf5e3ff675477cb9f9c874292fa63f [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:01.568498 28519 raft_consensus.cc:2258] T 27c72ce2560f4d3baf61f0441f5c4129 P 78bf5e3ff675477cb9f9c874292fa63f [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:01.570297 28519 tablet_replica.cc:330] T f4895041b0924276aa587adbb8dc9e98 P 78bf5e3ff675477cb9f9c874292fa63f: stopping tablet replica
I20240501 20:01:01.570732 28519 raft_consensus.cc:2229] T f4895041b0924276aa587adbb8dc9e98 P 78bf5e3ff675477cb9f9c874292fa63f [term 4 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:01.571130 28519 raft_consensus.cc:2258] T f4895041b0924276aa587adbb8dc9e98 P 78bf5e3ff675477cb9f9c874292fa63f [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:01.572814 28519 tablet_replica.cc:330] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f: stopping tablet replica
I20240501 20:01:01.573269 28519 raft_consensus.cc:2229] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f [term 3 LEADER]: Raft consensus shutting down.
I20240501 20:01:01.573969 28519 raft_consensus.cc:2258] T 5bd59b3e54144bd2986538a355cadd9b P 78bf5e3ff675477cb9f9c874292fa63f [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:01.575507 28519 tablet_replica.cc:330] T d057ab88d0084efe8d2fe408a42a24b7 P 78bf5e3ff675477cb9f9c874292fa63f: stopping tablet replica
I20240501 20:01:01.575953 28519 raft_consensus.cc:2229] T d057ab88d0084efe8d2fe408a42a24b7 P 78bf5e3ff675477cb9f9c874292fa63f [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:01.576371 28519 raft_consensus.cc:2258] T d057ab88d0084efe8d2fe408a42a24b7 P 78bf5e3ff675477cb9f9c874292fa63f [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:01.578032 28519 tablet_replica.cc:330] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 78bf5e3ff675477cb9f9c874292fa63f: stopping tablet replica
I20240501 20:01:01.578576 28519 raft_consensus.cc:2229] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 78bf5e3ff675477cb9f9c874292fa63f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:01.578953 28519 raft_consensus.cc:2258] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 78bf5e3ff675477cb9f9c874292fa63f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:01.580660 28519 tablet_replica.cc:330] T 3d51e8d55df849e5ac6e5acdc42ac63c P 78bf5e3ff675477cb9f9c874292fa63f: stopping tablet replica
I20240501 20:01:01.581104 28519 raft_consensus.cc:2229] T 3d51e8d55df849e5ac6e5acdc42ac63c P 78bf5e3ff675477cb9f9c874292fa63f [term 4 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:01.581552 28519 raft_consensus.cc:2258] T 3d51e8d55df849e5ac6e5acdc42ac63c P 78bf5e3ff675477cb9f9c874292fa63f [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:01.583091 28519 tablet_replica.cc:330] T eed1ac06c82147e3bb681f4bcd1d8688 P 78bf5e3ff675477cb9f9c874292fa63f: stopping tablet replica
I20240501 20:01:01.583575 28519 raft_consensus.cc:2229] T eed1ac06c82147e3bb681f4bcd1d8688 P 78bf5e3ff675477cb9f9c874292fa63f [term 2 LEADER]: Raft consensus shutting down.
I20240501 20:01:01.584347 28519 raft_consensus.cc:2258] T eed1ac06c82147e3bb681f4bcd1d8688 P 78bf5e3ff675477cb9f9c874292fa63f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:01.585889 28519 tablet_replica.cc:330] T afbcfea6a7894b5ab7c122d3d4134869 P 78bf5e3ff675477cb9f9c874292fa63f: stopping tablet replica
I20240501 20:01:01.586335 28519 raft_consensus.cc:2229] T afbcfea6a7894b5ab7c122d3d4134869 P 78bf5e3ff675477cb9f9c874292fa63f [term 3 LEADER]: Raft consensus shutting down.
I20240501 20:01:01.587054 28519 raft_consensus.cc:2258] T afbcfea6a7894b5ab7c122d3d4134869 P 78bf5e3ff675477cb9f9c874292fa63f [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:01.588627 28519 tablet_replica.cc:330] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 78bf5e3ff675477cb9f9c874292fa63f: stopping tablet replica
I20240501 20:01:01.589038 28519 raft_consensus.cc:2229] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 78bf5e3ff675477cb9f9c874292fa63f [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:01.589424 28519 raft_consensus.cc:2258] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 78bf5e3ff675477cb9f9c874292fa63f [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:01.617497 28519 tablet_server.cc:194] TabletServer@127.27.217.194:0 shutdown complete.
W20240501 20:01:01.632095 29273 consensus_peers.cc:485] T 419e173751fc4cd18b50341160a84d24 P 16f5b5441d8c4a44a9c0a6cca8356822 -> Peer 8feebcdef3ba428ba32e62016f3ed365 (127.27.217.193:43937): Couldn't send request to peer 8feebcdef3ba428ba32e62016f3ed365. Status: Network error: Client connection negotiation failed: client connection to 127.27.217.193:43937: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:01.637465 28519 tablet_server.cc:178] TabletServer@127.27.217.195:0 shutting down...
I20240501 20:01:01.662318 28519 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 20:01:01.662978 28519 tablet_replica.cc:330] T d057ab88d0084efe8d2fe408a42a24b7 P 16f5b5441d8c4a44a9c0a6cca8356822: stopping tablet replica
I20240501 20:01:01.663676 28519 raft_consensus.cc:2229] T d057ab88d0084efe8d2fe408a42a24b7 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:01.664294 28519 raft_consensus.cc:2258] T d057ab88d0084efe8d2fe408a42a24b7 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:01.666088 28519 tablet_replica.cc:330] T 419e173751fc4cd18b50341160a84d24 P 16f5b5441d8c4a44a9c0a6cca8356822: stopping tablet replica
I20240501 20:01:01.666617 28519 raft_consensus.cc:2229] T 419e173751fc4cd18b50341160a84d24 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 4 LEADER]: Raft consensus shutting down.
I20240501 20:01:01.667390 28519 raft_consensus.cc:2258] T 419e173751fc4cd18b50341160a84d24 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:01.669154 28519 tablet_replica.cc:330] T 5bd59b3e54144bd2986538a355cadd9b P 16f5b5441d8c4a44a9c0a6cca8356822: stopping tablet replica
I20240501 20:01:01.669639 28519 raft_consensus.cc:2229] T 5bd59b3e54144bd2986538a355cadd9b P 16f5b5441d8c4a44a9c0a6cca8356822 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:01.670053 28519 raft_consensus.cc:2258] T 5bd59b3e54144bd2986538a355cadd9b P 16f5b5441d8c4a44a9c0a6cca8356822 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:01.671638 28519 tablet_replica.cc:330] T 27c72ce2560f4d3baf61f0441f5c4129 P 16f5b5441d8c4a44a9c0a6cca8356822: stopping tablet replica
I20240501 20:01:01.672181 28519 raft_consensus.cc:2229] T 27c72ce2560f4d3baf61f0441f5c4129 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:01.672603 28519 raft_consensus.cc:2258] T 27c72ce2560f4d3baf61f0441f5c4129 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:01.674162 28519 tablet_replica.cc:330] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822: stopping tablet replica
I20240501 20:01:01.674643 28519 raft_consensus.cc:2229] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:01.675036 28519 raft_consensus.cc:2258] T f4895041b0924276aa587adbb8dc9e98 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:01.676731 28519 tablet_replica.cc:330] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 16f5b5441d8c4a44a9c0a6cca8356822: stopping tablet replica
I20240501 20:01:01.677184 28519 raft_consensus.cc:2229] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 16f5b5441d8c4a44a9c0a6cca8356822 [term 2 LEADER]: Raft consensus shutting down.
I20240501 20:01:01.677886 28519 raft_consensus.cc:2258] T 6c7b0c234f7d4cc0a0848e47e24c5d8b P 16f5b5441d8c4a44a9c0a6cca8356822 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:01.679347 28519 tablet_replica.cc:330] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822: stopping tablet replica
I20240501 20:01:01.679805 28519 raft_consensus.cc:2229] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:01.680225 28519 raft_consensus.cc:2258] T 3d51e8d55df849e5ac6e5acdc42ac63c P 16f5b5441d8c4a44a9c0a6cca8356822 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:01.681716 28519 tablet_replica.cc:330] T eed1ac06c82147e3bb681f4bcd1d8688 P 16f5b5441d8c4a44a9c0a6cca8356822: stopping tablet replica
I20240501 20:01:01.682183 28519 raft_consensus.cc:2229] T eed1ac06c82147e3bb681f4bcd1d8688 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:01.682569 28519 raft_consensus.cc:2258] T eed1ac06c82147e3bb681f4bcd1d8688 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:01.684154 28519 tablet_replica.cc:330] T afbcfea6a7894b5ab7c122d3d4134869 P 16f5b5441d8c4a44a9c0a6cca8356822: stopping tablet replica
I20240501 20:01:01.684587 28519 raft_consensus.cc:2229] T afbcfea6a7894b5ab7c122d3d4134869 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:01.684976 28519 raft_consensus.cc:2258] T afbcfea6a7894b5ab7c122d3d4134869 P 16f5b5441d8c4a44a9c0a6cca8356822 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:01.686707 28519 tablet_replica.cc:330] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 16f5b5441d8c4a44a9c0a6cca8356822: stopping tablet replica
I20240501 20:01:01.687109 28519 raft_consensus.cc:2229] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 16f5b5441d8c4a44a9c0a6cca8356822 [term 3 LEADER]: Raft consensus shutting down.
I20240501 20:01:01.687829 28519 raft_consensus.cc:2258] T 0d9b61e9c98e4d19b9272fe70fab0c3d P 16f5b5441d8c4a44a9c0a6cca8356822 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:01.715608 28519 tablet_server.cc:194] TabletServer@127.27.217.195:0 shutdown complete.
I20240501 20:01:01.733290 28519 master.cc:537] Master@127.27.217.254:34681 shutting down...
I20240501 20:01:01.755009 28519 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e673238763d6498fbb9641c1f1bcb503 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:01:01.755561 28519 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e673238763d6498fbb9641c1f1bcb503 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:01.755872 28519 tablet_replica.cc:330] T 00000000000000000000000000000000 P e673238763d6498fbb9641c1f1bcb503: stopping tablet replica
I20240501 20:01:01.773757 28519 master.cc:552] Master@127.27.217.254:34681 shutdown complete.
16f5b5441d8c4a44a9c0a6cca8356822 1
78bf5e3ff675477cb9f9c874292fa63f 4
8feebcdef3ba428ba32e62016f3ed365 5
16f5b5441d8c4a44a9c0a6cca8356822 3
78bf5e3ff675477cb9f9c874292fa63f 4
8feebcdef3ba428ba32e62016f3ed365 3
16f5b5441d8c4a44a9c0a6cca8356822 1
78bf5e3ff675477cb9f9c874292fa63f 1
8feebcdef3ba428ba32e62016f3ed365 8
16f5b5441d8c4a44a9c0a6cca8356822 3
78bf5e3ff675477cb9f9c874292fa63f 3
8feebcdef3ba428ba32e62016f3ed365 4
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (28525 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20240501 20:01:01.816499 28519 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.217.254:42703
I20240501 20:01:01.817625 28519 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:01.823088 29533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:01.824456 29534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:01.824734 29536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:01.824800 28519 server_base.cc:1034] running on GCE node
I20240501 20:01:01.826045 28519 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:01:01.826243 28519 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:01:01.826360 28519 hybrid_clock.cc:648] HybridClock initialized: now 1714593661826349 us; error 0 us; skew 500 ppm
I20240501 20:01:01.826820 28519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:01:01.832772 28519 webserver.cc:460] Webserver started at http://127.27.217.254:44113/ using document root <none> and password file <none>
I20240501 20:01:01.833237 28519 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:01.833407 28519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:01.833642 28519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:01.834817 28519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714593602638719-28519-0/minicluster-data/master-0-root/instance:
uuid: "995802798c1e438daecd368538bebecd"
format_stamp: "Formatted at 2024-05-01 20:01:01 on dist-test-slave-92m0"
I20240501 20:01:01.839202 28519 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240501 20:01:01.842435 29541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:01.843142 28519 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:01:01.843420 28519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714593602638719-28519-0/minicluster-data/master-0-root
uuid: "995802798c1e438daecd368538bebecd"
format_stamp: "Formatted at 2024-05-01 20:01:01 on dist-test-slave-92m0"
I20240501 20:01:01.843698 28519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714593602638719-28519-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714593602638719-28519-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714593602638719-28519-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:01.861678 28519 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:01.862911 28519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:01.898828 28519 rpc_server.cc:307] RPC server started. Bound to: 127.27.217.254:42703
I20240501 20:01:01.898924 29592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.217.254:42703 every 8 connection(s)
I20240501 20:01:01.902639 29593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:01.912818 29593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 995802798c1e438daecd368538bebecd: Bootstrap starting.
I20240501 20:01:01.916805 29593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 995802798c1e438daecd368538bebecd: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:01.920814 29593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 995802798c1e438daecd368538bebecd: No bootstrap required, opened a new log
I20240501 20:01:01.922850 29593 raft_consensus.cc:348] T 00000000000000000000000000000000 P 995802798c1e438daecd368538bebecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "995802798c1e438daecd368538bebecd" member_type: VOTER }
I20240501 20:01:01.923386 29593 raft_consensus.cc:374] T 00000000000000000000000000000000 P 995802798c1e438daecd368538bebecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:01.923591 29593 raft_consensus.cc:729] T 00000000000000000000000000000000 P 995802798c1e438daecd368538bebecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 995802798c1e438daecd368538bebecd, State: Initialized, Role: FOLLOWER
I20240501 20:01:01.924041 29593 consensus_queue.cc:260] T 00000000000000000000000000000000 P 995802798c1e438daecd368538bebecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "995802798c1e438daecd368538bebecd" member_type: VOTER }
I20240501 20:01:01.924486 29593 raft_consensus.cc:388] T 00000000000000000000000000000000 P 995802798c1e438daecd368538bebecd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:01:01.924693 29593 raft_consensus.cc:482] T 00000000000000000000000000000000 P 995802798c1e438daecd368538bebecd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:01:01.924902 29593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 995802798c1e438daecd368538bebecd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:01.928715 29593 raft_consensus.cc:504] T 00000000000000000000000000000000 P 995802798c1e438daecd368538bebecd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "995802798c1e438daecd368538bebecd" member_type: VOTER }
I20240501 20:01:01.929150 29593 leader_election.cc:302] T 00000000000000000000000000000000 P 995802798c1e438daecd368538bebecd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 995802798c1e438daecd368538bebecd; no voters:
I20240501 20:01:01.930172 29593 leader_election.cc:288] T 00000000000000000000000000000000 P 995802798c1e438daecd368538bebecd [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:01:01.930577 29596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 995802798c1e438daecd368538bebecd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:01.931897 29596 raft_consensus.cc:686] T 00000000000000000000000000000000 P 995802798c1e438daecd368538bebecd [term 1 LEADER]: Becoming Leader. State: Replica: 995802798c1e438daecd368538bebecd, State: Running, Role: LEADER
I20240501 20:01:01.932675 29596 consensus_queue.cc:237] T 00000000000000000000000000000000 P 995802798c1e438daecd368538bebecd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "995802798c1e438daecd368538bebecd" member_type: VOTER }
I20240501 20:01:01.933099 29593 sys_catalog.cc:564] T 00000000000000000000000000000000 P 995802798c1e438daecd368538bebecd [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:01:01.937209 29597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 995802798c1e438daecd368538bebecd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "995802798c1e438daecd368538bebecd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "995802798c1e438daecd368538bebecd" member_type: VOTER } }
I20240501 20:01:01.937503 29598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 995802798c1e438daecd368538bebecd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 995802798c1e438daecd368538bebecd. Latest consensus state: current_term: 1 leader_uuid: "995802798c1e438daecd368538bebecd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "995802798c1e438daecd368538bebecd" member_type: VOTER } }
I20240501 20:01:01.937903 29597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 995802798c1e438daecd368538bebecd [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:01.938230 29598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 995802798c1e438daecd368538bebecd [sys.catalog]: This master's current role is: LEADER
I20240501 20:01:01.941397 29602 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:01:01.949527 29602 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:01:01.950609 28519 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 20:01:01.957875 29602 catalog_manager.cc:1347] Generated new cluster ID: 9988be11e9e44272a8f2bf0337c21fd1
I20240501 20:01:01.958179 29602 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:01:01.986279 29602 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:01:01.987474 29602 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:01:02.002875 29602 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 995802798c1e438daecd368538bebecd: Generated new TSK 0
I20240501 20:01:02.003504 29602 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:01:02.017616 28519 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:02.023017 29614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:02.024566 29615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:02.025991 28519 server_base.cc:1034] running on GCE node
W20240501 20:01:02.026039 29617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:02.026947 28519 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:01:02.027156 28519 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:01:02.027321 28519 hybrid_clock.cc:648] HybridClock initialized: now 1714593662027304 us; error 0 us; skew 500 ppm
I20240501 20:01:02.027834 28519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:01:02.030114 28519 webserver.cc:460] Webserver started at http://127.27.217.193:38839/ using document root <none> and password file <none>
I20240501 20:01:02.030583 28519 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:02.030776 28519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:02.031028 28519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:02.032132 28519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714593602638719-28519-0/minicluster-data/ts-0-root/instance:
uuid: "4c8da1d580614449a636b855bf11b95f"
format_stamp: "Formatted at 2024-05-01 20:01:02 on dist-test-slave-92m0"
I20240501 20:01:02.036754 28519 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20240501 20:01:02.039891 29622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:02.040637 28519 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 20:01:02.040899 28519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714593602638719-28519-0/minicluster-data/ts-0-root
uuid: "4c8da1d580614449a636b855bf11b95f"
format_stamp: "Formatted at 2024-05-01 20:01:02 on dist-test-slave-92m0"
I20240501 20:01:02.041180 28519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714593602638719-28519-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714593602638719-28519-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714593602638719-28519-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:02.063072 28519 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:02.064333 28519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:02.065785 28519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:01:02.067975 28519 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:01:02.068207 28519 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:02.068434 28519 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:01:02.068605 28519 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:02.105831 28519 rpc_server.cc:307] RPC server started. Bound to: 127.27.217.193:33959
I20240501 20:01:02.105912 29684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.217.193:33959 every 8 connection(s)
I20240501 20:01:02.110152 28519 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:02.117746 29689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:02.120343 29690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:02.124092 29692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:02.124222 29685 heartbeater.cc:347] Connected to a master server at 127.27.217.254:42703
I20240501 20:01:02.124713 29685 heartbeater.cc:464] Registering TS with master...
I20240501 20:01:02.125383 29685 heartbeater.cc:511] Master 127.27.217.254:42703 requested a full tablet report, sending...
I20240501 20:01:02.125651 28519 server_base.cc:1034] running on GCE node
I20240501 20:01:02.126590 28519 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:01:02.126791 28519 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:01:02.126955 28519 hybrid_clock.cc:648] HybridClock initialized: now 1714593662126934 us; error 0 us; skew 500 ppm
I20240501 20:01:02.127298 29558 ts_manager.cc:194] Registered new tserver with Master: 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959)
I20240501 20:01:02.127480 28519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:01:02.129416 29558 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43370
I20240501 20:01:02.130285 28519 webserver.cc:460] Webserver started at http://127.27.217.194:33677/ using document root <none> and password file <none>
I20240501 20:01:02.130846 28519 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:02.131050 28519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:02.131296 28519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:02.132469 28519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714593602638719-28519-0/minicluster-data/ts-1-root/instance:
uuid: "fdfeac4190e045f290e003b2298d988d"
format_stamp: "Formatted at 2024-05-01 20:01:02 on dist-test-slave-92m0"
I20240501 20:01:02.136731 28519 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240501 20:01:02.139777 29697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:02.140489 28519 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 20:01:02.140753 28519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714593602638719-28519-0/minicluster-data/ts-1-root
uuid: "fdfeac4190e045f290e003b2298d988d"
format_stamp: "Formatted at 2024-05-01 20:01:02 on dist-test-slave-92m0"
I20240501 20:01:02.141041 28519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714593602638719-28519-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714593602638719-28519-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714593602638719-28519-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:02.160854 28519 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:02.162051 28519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:02.163369 28519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:01:02.165546 28519 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:01:02.165742 28519 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:02.165979 28519 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:01:02.166136 28519 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:02.208425 28519 rpc_server.cc:307] RPC server started. Bound to: 127.27.217.194:32907
I20240501 20:01:02.208524 29759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.217.194:32907 every 8 connection(s)
I20240501 20:01:02.213003 28519 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:02.219322 29763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:02.219802 29764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:02.224607 29766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:02.225154 28519 server_base.cc:1034] running on GCE node
I20240501 20:01:02.225911 28519 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240501 20:01:02.226009 29760 heartbeater.cc:347] Connected to a master server at 127.27.217.254:42703
W20240501 20:01:02.226190 28519 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:01:02.226413 29760 heartbeater.cc:464] Registering TS with master...
I20240501 20:01:02.226413 28519 hybrid_clock.cc:648] HybridClock initialized: now 1714593662226391 us; error 0 us; skew 500 ppm
I20240501 20:01:02.227382 28519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:01:02.227387 29760 heartbeater.cc:511] Master 127.27.217.254:42703 requested a full tablet report, sending...
I20240501 20:01:02.229729 29558 ts_manager.cc:194] Registered new tserver with Master: fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:02.231076 29558 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43384
I20240501 20:01:02.232185 28519 webserver.cc:460] Webserver started at http://127.27.217.195:38311/ using document root <none> and password file <none>
I20240501 20:01:02.232829 28519 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:02.233086 28519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:02.233419 28519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:02.234601 28519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714593602638719-28519-0/minicluster-data/ts-2-root/instance:
uuid: "ee0d97247d394a8facfa774f15f3a67d"
format_stamp: "Formatted at 2024-05-01 20:01:02 on dist-test-slave-92m0"
I20240501 20:01:02.238832 28519 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240501 20:01:02.242431 29771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:02.243274 28519 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 20:01:02.243539 28519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714593602638719-28519-0/minicluster-data/ts-2-root
uuid: "ee0d97247d394a8facfa774f15f3a67d"
format_stamp: "Formatted at 2024-05-01 20:01:02 on dist-test-slave-92m0"
I20240501 20:01:02.243832 28519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714593602638719-28519-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714593602638719-28519-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714593602638719-28519-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:02.256880 28519 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:02.258077 28519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:02.259449 28519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:01:02.261634 28519 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:01:02.261840 28519 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:02.262066 28519 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:01:02.262224 28519 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:02.298983 28519 rpc_server.cc:307] RPC server started. Bound to: 127.27.217.195:35861
I20240501 20:01:02.299074 29833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.217.195:35861 every 8 connection(s)
I20240501 20:01:02.311650 29834 heartbeater.cc:347] Connected to a master server at 127.27.217.254:42703
I20240501 20:01:02.312004 29834 heartbeater.cc:464] Registering TS with master...
I20240501 20:01:02.312727 29834 heartbeater.cc:511] Master 127.27.217.254:42703 requested a full tablet report, sending...
I20240501 20:01:02.314409 29558 ts_manager.cc:194] Registered new tserver with Master: ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:02.315065 28519 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013210755s
I20240501 20:01:02.315634 29558 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43396
I20240501 20:01:02.317536 28519 test_util.cc:268] Using random seed: -231918645
I20240501 20:01:02.338939 29558 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43398:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 20:01:02.345568 29558 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 20:01:02.549168 29798 tablet_service.cc:1467] Processing CreateTablet for tablet d1d2cac25f354a97b1bfb53d57ae7c08 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20240501 20:01:02.550639 29798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1d2cac25f354a97b1bfb53d57ae7c08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.551178 29790 tablet_service.cc:1467] Processing CreateTablet for tablet 1ca3948261374cef8b1b206a43c00173 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20240501 20:01:02.552492 29790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ca3948261374cef8b1b206a43c00173. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.553208 29797 tablet_service.cc:1467] Processing CreateTablet for tablet bfac893b4b4f457694a4eb184b6abb53 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20240501 20:01:02.554486 29797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfac893b4b4f457694a4eb184b6abb53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.557195 29796 tablet_service.cc:1467] Processing CreateTablet for tablet cb05df43bdfa479b902b1da297fb5974 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20240501 20:01:02.558488 29796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb05df43bdfa479b902b1da297fb5974. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.560035 29791 tablet_service.cc:1467] Processing CreateTablet for tablet add9d7579351483fb268f89a44e95d8a (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20240501 20:01:02.561367 29791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet add9d7579351483fb268f89a44e95d8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.563647 29725 tablet_service.cc:1467] Processing CreateTablet for tablet 69915dd5992641e3a24f7f45f1cd5c0c (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION VALUES < 36398027
I20240501 20:01:02.564970 29725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69915dd5992641e3a24f7f45f1cd5c0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.565150 29792 tablet_service.cc:1467] Processing CreateTablet for tablet 49da4246f5f6438abe0e7e8c9a523d80 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20240501 20:01:02.566430 29792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49da4246f5f6438abe0e7e8c9a523d80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.569145 29793 tablet_service.cc:1467] Processing CreateTablet for tablet 7a3f29f0eeea4b119984634202e5d9cd (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20240501 20:01:02.570432 29793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a3f29f0eeea4b119984634202e5d9cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.573227 29794 tablet_service.cc:1467] Processing CreateTablet for tablet 9046e38fc57f4364aa3d539e1fc01928 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20240501 20:01:02.574501 29794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9046e38fc57f4364aa3d539e1fc01928. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.577205 29795 tablet_service.cc:1467] Processing CreateTablet for tablet 668b4a4d782c4278932dd8413dc3ab3e (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20240501 20:01:02.578474 29795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 668b4a4d782c4278932dd8413dc3ab3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.547765 29799 tablet_service.cc:1467] Processing CreateTablet for tablet 69915dd5992641e3a24f7f45f1cd5c0c (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION VALUES < 36398027
I20240501 20:01:02.581027 29849 tablet_bootstrap.cc:492] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:02.581354 29799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69915dd5992641e3a24f7f45f1cd5c0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.582872 29796 tablet_service.cc:1467] Processing CreateTablet for tablet f6d88139f89c47ff84cad674147aae0f (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20240501 20:01:02.582952 29722 tablet_service.cc:1467] Processing CreateTablet for tablet cb05df43bdfa479b902b1da297fb5974 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20240501 20:01:02.584214 29796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6d88139f89c47ff84cad674147aae0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.587126 29723 tablet_service.cc:1467] Processing CreateTablet for tablet bfac893b4b4f457694a4eb184b6abb53 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20240501 20:01:02.587388 29717 tablet_service.cc:1467] Processing CreateTablet for tablet add9d7579351483fb268f89a44e95d8a (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20240501 20:01:02.589032 29720 tablet_service.cc:1467] Processing CreateTablet for tablet 9046e38fc57f4364aa3d539e1fc01928 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20240501 20:01:02.590617 29716 tablet_service.cc:1467] Processing CreateTablet for tablet 1ca3948261374cef8b1b206a43c00173 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20240501 20:01:02.591045 29724 tablet_service.cc:1467] Processing CreateTablet for tablet d1d2cac25f354a97b1bfb53d57ae7c08 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20240501 20:01:02.591711 29849 tablet_bootstrap.cc:654] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:02.593041 29718 tablet_service.cc:1467] Processing CreateTablet for tablet 49da4246f5f6438abe0e7e8c9a523d80 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20240501 20:01:02.594977 29721 tablet_service.cc:1467] Processing CreateTablet for tablet 668b4a4d782c4278932dd8413dc3ab3e (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20240501 20:01:02.596935 29719 tablet_service.cc:1467] Processing CreateTablet for tablet 7a3f29f0eeea4b119984634202e5d9cd (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20240501 20:01:02.584321 29722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb05df43bdfa479b902b1da297fb5974. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.599712 29717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet add9d7579351483fb268f89a44e95d8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.600360 29719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a3f29f0eeea4b119984634202e5d9cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.600943 29718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49da4246f5f6438abe0e7e8c9a523d80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.601641 29721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 668b4a4d782c4278932dd8413dc3ab3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.604398 29720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9046e38fc57f4364aa3d539e1fc01928. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.608418 29723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfac893b4b4f457694a4eb184b6abb53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.611001 29796 tablet_service.cc:1467] Processing CreateTablet for tablet 2d86f8848a184a87947fd927c44e5a35 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20240501 20:01:02.612287 29796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d86f8848a184a87947fd927c44e5a35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.612424 29724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1d2cac25f354a97b1bfb53d57ae7c08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.619093 29716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ca3948261374cef8b1b206a43c00173. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.651561 29650 tablet_service.cc:1467] Processing CreateTablet for tablet 69915dd5992641e3a24f7f45f1cd5c0c (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION VALUES < 36398027
I20240501 20:01:02.653327 29650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69915dd5992641e3a24f7f45f1cd5c0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.657703 29649 tablet_service.cc:1467] Processing CreateTablet for tablet d1d2cac25f354a97b1bfb53d57ae7c08 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20240501 20:01:02.659492 29649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1d2cac25f354a97b1bfb53d57ae7c08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.661682 29648 tablet_service.cc:1467] Processing CreateTablet for tablet bfac893b4b4f457694a4eb184b6abb53 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20240501 20:01:02.663430 29648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfac893b4b4f457694a4eb184b6abb53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.665601 29647 tablet_service.cc:1467] Processing CreateTablet for tablet cb05df43bdfa479b902b1da297fb5974 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20240501 20:01:02.667338 29647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb05df43bdfa479b902b1da297fb5974. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.669610 29646 tablet_service.cc:1467] Processing CreateTablet for tablet 668b4a4d782c4278932dd8413dc3ab3e (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20240501 20:01:02.671317 29646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 668b4a4d782c4278932dd8413dc3ab3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.673595 29645 tablet_service.cc:1467] Processing CreateTablet for tablet 9046e38fc57f4364aa3d539e1fc01928 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20240501 20:01:02.675307 29645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9046e38fc57f4364aa3d539e1fc01928. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.678745 29644 tablet_service.cc:1467] Processing CreateTablet for tablet 7a3f29f0eeea4b119984634202e5d9cd (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20240501 20:01:02.682811 29643 tablet_service.cc:1467] Processing CreateTablet for tablet 49da4246f5f6438abe0e7e8c9a523d80 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20240501 20:01:02.683426 29794 tablet_service.cc:1467] Processing CreateTablet for tablet bac2171683884f6f97cdfeb0a656cb49 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20240501 20:01:02.686302 29642 tablet_service.cc:1467] Processing CreateTablet for tablet add9d7579351483fb268f89a44e95d8a (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20240501 20:01:02.689144 29641 tablet_service.cc:1467] Processing CreateTablet for tablet 1ca3948261374cef8b1b206a43c00173 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20240501 20:01:02.699810 29644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a3f29f0eeea4b119984634202e5d9cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.720366 29641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ca3948261374cef8b1b206a43c00173. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.720791 29723 tablet_service.cc:1467] Processing CreateTablet for tablet 8bd5e83faaff4e2da4e4cc22ba410d33 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20240501 20:01:02.722034 29723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bd5e83faaff4e2da4e4cc22ba410d33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.724588 29642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet add9d7579351483fb268f89a44e95d8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.678944 29717 tablet_service.cc:1467] Processing CreateTablet for tablet f6d88139f89c47ff84cad674147aae0f (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20240501 20:01:02.711289 29724 tablet_service.cc:1467] Processing CreateTablet for tablet 2d86f8848a184a87947fd927c44e5a35 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20240501 20:01:02.731782 29794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bac2171683884f6f97cdfeb0a656cb49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.732856 29720 tablet_service.cc:1467] Processing CreateTablet for tablet 696121417ea9403990456cfc6fc4f416 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20240501 20:01:02.703612 29718 tablet_service.cc:1467] Processing CreateTablet for tablet bac2171683884f6f97cdfeb0a656cb49 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20240501 20:01:02.742213 29791 tablet_service.cc:1467] Processing CreateTablet for tablet 8bd5e83faaff4e2da4e4cc22ba410d33 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20240501 20:01:02.744330 29791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bd5e83faaff4e2da4e4cc22ba410d33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.763303 29725 tablet_service.cc:1467] Processing CreateTablet for tablet 6e38a2e91a3d48d09a22abce723af833 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20240501 20:01:02.772161 29851 tablet_bootstrap.cc:492] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:02.771703 29643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49da4246f5f6438abe0e7e8c9a523d80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.742161 29790 tablet_service.cc:1467] Processing CreateTablet for tablet 696121417ea9403990456cfc6fc4f416 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20240501 20:01:02.796277 29719 tablet_service.cc:1467] Processing CreateTablet for tablet be383be783234855878cb8fd442c7b05 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20240501 20:01:02.798905 29792 tablet_service.cc:1467] Processing CreateTablet for tablet 9f261d2a05bf4c838532049601694b4a (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20240501 20:01:02.800315 29792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f261d2a05bf4c838532049601694b4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.805955 29795 tablet_service.cc:1467] Processing CreateTablet for tablet 6e38a2e91a3d48d09a22abce723af833 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20240501 20:01:02.807300 29795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e38a2e91a3d48d09a22abce723af833. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.776638 29724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d86f8848a184a87947fd927c44e5a35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.815641 29849 tablet_bootstrap.cc:492] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:02.810300 29722 tablet_service.cc:1467] Processing CreateTablet for tablet 649b71f9f267437ca51c758dd9ca6987 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20240501 20:01:02.816169 29849 ts_tablet_manager.cc:1367] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.235s user 0.063s sys 0.080s
I20240501 20:01:02.816872 29725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e38a2e91a3d48d09a22abce723af833. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.775934 29721 tablet_service.cc:1467] Processing CreateTablet for tablet 9f261d2a05bf4c838532049601694b4a (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20240501 20:01:02.821111 29721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f261d2a05bf4c838532049601694b4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.846051 29719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be383be783234855878cb8fd442c7b05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.847254 29718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bac2171683884f6f97cdfeb0a656cb49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.847744 29849 raft_consensus.cc:348] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:02.849658 29790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 696121417ea9403990456cfc6fc4f416. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.850824 29720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 696121417ea9403990456cfc6fc4f416. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.855336 29798 tablet_service.cc:1467] Processing CreateTablet for tablet 48630ae6eb394ac1b54955dd01cf0ed2 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20240501 20:01:02.856854 29717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6d88139f89c47ff84cad674147aae0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.863447 29798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48630ae6eb394ac1b54955dd01cf0ed2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.851526 29796 tablet_service.cc:1467] Processing CreateTablet for tablet be383be783234855878cb8fd442c7b05 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20240501 20:01:02.860865 29849 raft_consensus.cc:374] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:02.876663 29849 raft_consensus.cc:729] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:02.877449 29849 consensus_queue.cc:260] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:02.880182 29796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be383be783234855878cb8fd442c7b05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.890612 29799 tablet_service.cc:1467] Processing CreateTablet for tablet 38cbeb0dc0c14e3d9182101eea885625 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20240501 20:01:02.860236 29797 tablet_service.cc:1467] Processing CreateTablet for tablet 649b71f9f267437ca51c758dd9ca6987 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20240501 20:01:02.872279 29722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 649b71f9f267437ca51c758dd9ca6987. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.873836 29851 tablet_bootstrap.cc:654] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:02.898753 29720 tablet_service.cc:1467] Processing CreateTablet for tablet 48630ae6eb394ac1b54955dd01cf0ed2 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20240501 20:01:02.900081 29720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48630ae6eb394ac1b54955dd01cf0ed2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.916836 29852 tablet_bootstrap.cc:492] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:02.891899 29799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38cbeb0dc0c14e3d9182101eea885625. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.924683 29725 tablet_service.cc:1467] Processing CreateTablet for tablet 38cbeb0dc0c14e3d9182101eea885625 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20240501 20:01:02.925670 29797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 649b71f9f267437ca51c758dd9ca6987. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.939522 29650 tablet_service.cc:1467] Processing CreateTablet for tablet f6d88139f89c47ff84cad674147aae0f (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20240501 20:01:02.940917 29650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6d88139f89c47ff84cad674147aae0f. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 20:01:02.946389 29606 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240501 20:01:02.947089 29645 tablet_service.cc:1467] Processing CreateTablet for tablet bac2171683884f6f97cdfeb0a656cb49 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20240501 20:01:02.949265 29644 tablet_service.cc:1467] Processing CreateTablet for tablet 8bd5e83faaff4e2da4e4cc22ba410d33 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20240501 20:01:02.950464 29644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bd5e83faaff4e2da4e4cc22ba410d33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.951799 29794 tablet_service.cc:1467] Processing CreateTablet for tablet 456035a1cb3f4f7da52d0fc1dd1c656c (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20240501 20:01:02.953351 29794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 456035a1cb3f4f7da52d0fc1dd1c656c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.955138 29791 tablet_service.cc:1467] Processing CreateTablet for tablet 9472c53f79ec423f82c396bd980674e2 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20240501 20:01:02.956647 29791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9472c53f79ec423f82c396bd980674e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.958600 29648 tablet_service.cc:1467] Processing CreateTablet for tablet 696121417ea9403990456cfc6fc4f416 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20240501 20:01:02.961359 29793 tablet_service.cc:1467] Processing CreateTablet for tablet 3af08384057a4d85977481472f460168 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20240501 20:01:02.962613 29793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3af08384057a4d85977481472f460168. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.965349 29790 tablet_service.cc:1467] Processing CreateTablet for tablet bbcf30cf66714efdac997862dafbe4a2 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20240501 20:01:02.966521 29790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbcf30cf66714efdac997862dafbe4a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.974682 29650 tablet_service.cc:1467] Processing CreateTablet for tablet 6e38a2e91a3d48d09a22abce723af833 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20240501 20:01:02.984948 29649 tablet_service.cc:1467] Processing CreateTablet for tablet 2d86f8848a184a87947fd927c44e5a35 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20240501 20:01:02.991602 29648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 696121417ea9403990456cfc6fc4f416. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.996798 29834 heartbeater.cc:503] Master 127.27.217.254:42703 was elected leader, sending a full tablet report...
I20240501 20:01:02.997503 29643 tablet_service.cc:1467] Processing CreateTablet for tablet 48630ae6eb394ac1b54955dd01cf0ed2 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20240501 20:01:02.998744 29643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48630ae6eb394ac1b54955dd01cf0ed2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.999162 29725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38cbeb0dc0c14e3d9182101eea885625. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:02.975670 29642 tablet_service.cc:1467] Processing CreateTablet for tablet be383be783234855878cb8fd442c7b05 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20240501 20:01:02.972023 29641 tablet_service.cc:1467] Processing CreateTablet for tablet 9f261d2a05bf4c838532049601694b4a (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20240501 20:01:03.018702 29849 ts_tablet_manager.cc:1398] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.201s user 0.022s sys 0.025s
I20240501 20:01:03.019959 29849 tablet_bootstrap.cc:492] T f6d88139f89c47ff84cad674147aae0f P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:03.022377 29716 tablet_service.cc:1467] Processing CreateTablet for tablet 456035a1cb3f4f7da52d0fc1dd1c656c (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20240501 20:01:03.023710 29716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 456035a1cb3f4f7da52d0fc1dd1c656c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.030359 29723 tablet_service.cc:1467] Processing CreateTablet for tablet 3af08384057a4d85977481472f460168 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20240501 20:01:03.031661 29723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3af08384057a4d85977481472f460168. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.032747 29796 tablet_service.cc:1467] Processing CreateTablet for tablet a7c597f15e844f30837012baed4b0501 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20240501 20:01:03.034188 29796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7c597f15e844f30837012baed4b0501. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.036563 29718 tablet_service.cc:1467] Processing CreateTablet for tablet 9472c53f79ec423f82c396bd980674e2 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20240501 20:01:03.039608 29851 tablet_bootstrap.cc:492] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:03.040359 29851 ts_tablet_manager.cc:1367] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.272s user 0.019s sys 0.009s
I20240501 20:01:03.040819 29717 tablet_service.cc:1467] Processing CreateTablet for tablet bbcf30cf66714efdac997862dafbe4a2 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20240501 20:01:03.041858 29718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9472c53f79ec423f82c396bd980674e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.017477 29650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e38a2e91a3d48d09a22abce723af833. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.043543 29649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d86f8848a184a87947fd927c44e5a35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.058552 29719 tablet_service.cc:1467] Processing CreateTablet for tablet 0fb27193abbf4644a48662909bc7b04a (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20240501 20:01:03.131290 29645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bac2171683884f6f97cdfeb0a656cb49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.136857 29685 heartbeater.cc:503] Master 127.27.217.254:42703 was elected leader, sending a full tablet report...
I20240501 20:01:03.137104 29641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f261d2a05bf4c838532049601694b4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.013921 29646 tablet_service.cc:1467] Processing CreateTablet for tablet 649b71f9f267437ca51c758dd9ca6987 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20240501 20:01:03.138995 29642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be383be783234855878cb8fd442c7b05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.140065 29646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 649b71f9f267437ca51c758dd9ca6987. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.044947 29717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbcf30cf66714efdac997862dafbe4a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.143601 29719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fb27193abbf4644a48662909bc7b04a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.045267 29792 tablet_service.cc:1467] Processing CreateTablet for tablet 0fb27193abbf4644a48662909bc7b04a (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20240501 20:01:03.148464 29792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fb27193abbf4644a48662909bc7b04a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.076298 29798 tablet_service.cc:1467] Processing CreateTablet for tablet 458740ab5c444b6ab3d1991c69db5084 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
W20240501 20:01:03.065407 29835 tablet.cc:2369] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:01:03.153846 29798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 458740ab5c444b6ab3d1991c69db5084. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.075807 29849 tablet_bootstrap.cc:654] T f6d88139f89c47ff84cad674147aae0f P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:03.077387 29795 tablet_service.cc:1467] Processing CreateTablet for tablet 010ceabf3e5a4b1c9330b80dade765bb (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20240501 20:01:03.162411 29795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 010ceabf3e5a4b1c9330b80dade765bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.077061 29797 tablet_service.cc:1467] Processing CreateTablet for tablet 07bdb8b09b8c4ef4b8e49daa9b8c01d6 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20240501 20:01:03.165338 29797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07bdb8b09b8c4ef4b8e49daa9b8c01d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.080780 29851 raft_consensus.cc:348] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } }
I20240501 20:01:03.167682 29851 raft_consensus.cc:374] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:03.168300 29851 raft_consensus.cc:729] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:03.168551 29647 tablet_service.cc:1467] Processing CreateTablet for tablet 38cbeb0dc0c14e3d9182101eea885625 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20240501 20:01:03.170671 29647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38cbeb0dc0c14e3d9182101eea885625. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.169498 29851 consensus_queue.cc:260] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } }
I20240501 20:01:03.069533 29852 tablet_bootstrap.cc:654] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:03.199985 29721 tablet_service.cc:1467] Processing CreateTablet for tablet a7c597f15e844f30837012baed4b0501 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20240501 20:01:03.201501 29721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7c597f15e844f30837012baed4b0501. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.214004 29643 tablet_service.cc:1467] Processing CreateTablet for tablet 0374bb790ed345ed90452595f71383a2 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20240501 20:01:03.229700 29722 tablet_service.cc:1467] Processing CreateTablet for tablet 07bdb8b09b8c4ef4b8e49daa9b8c01d6 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20240501 20:01:03.231065 29722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07bdb8b09b8c4ef4b8e49daa9b8c01d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.234033 29760 heartbeater.cc:503] Master 127.27.217.254:42703 was elected leader, sending a full tablet report...
I20240501 20:01:03.242141 29799 tablet_service.cc:1467] Processing CreateTablet for tablet b337b8e3e38d4ad6b1076fbd0d0a1609 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20240501 20:01:03.244246 29641 tablet_service.cc:1467] Processing CreateTablet for tablet 9472c53f79ec423f82c396bd980674e2 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20240501 20:01:03.245559 29641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9472c53f79ec423f82c396bd980674e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.216176 29724 tablet_service.cc:1467] Processing CreateTablet for tablet 458740ab5c444b6ab3d1991c69db5084 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20240501 20:01:03.261772 29724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 458740ab5c444b6ab3d1991c69db5084. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.220674 29650 tablet_service.cc:1467] Processing CreateTablet for tablet 456035a1cb3f4f7da52d0fc1dd1c656c (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20240501 20:01:03.244756 29716 tablet_service.cc:1467] Processing CreateTablet for tablet 010ceabf3e5a4b1c9330b80dade765bb (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20240501 20:01:03.267495 29716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 010ceabf3e5a4b1c9330b80dade765bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.264948 29799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b337b8e3e38d4ad6b1076fbd0d0a1609. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.275465 29791 tablet_service.cc:1467] Processing CreateTablet for tablet a8e9f75387564275a60d38e9a910319c (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20240501 20:01:03.259568 29790 tablet_service.cc:1467] Processing CreateTablet for tablet 77be7101c0fe4085b6706296f609870a (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20240501 20:01:03.295414 29790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77be7101c0fe4085b6706296f609870a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.267495 29650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 456035a1cb3f4f7da52d0fc1dd1c656c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.304839 29791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8e9f75387564275a60d38e9a910319c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.283458 29798 tablet_service.cc:1467] Processing CreateTablet for tablet 211e27a3b0084b89a6dd7550849b5a65 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20240501 20:01:03.307775 29798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 211e27a3b0084b89a6dd7550849b5a65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.289638 29793 tablet_service.cc:1467] Processing CreateTablet for tablet c651044a305c4168805825a53d910899 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20240501 20:01:03.312801 29793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c651044a305c4168805825a53d910899. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.314318 29794 tablet_service.cc:1467] Processing CreateTablet for tablet d48fd0a8e289401aa7417c42ff2c81b2 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20240501 20:01:03.315639 29794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d48fd0a8e289401aa7417c42ff2c81b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.316556 29645 tablet_service.cc:1467] Processing CreateTablet for tablet 3c1a08b455b94ae089396a15c837ddc2 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20240501 20:01:03.340838 29643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0374bb790ed345ed90452595f71383a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.344790 29645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c1a08b455b94ae089396a15c837ddc2. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 20:01:03.345448 29544 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 54222.7us
I20240501 20:01:03.348893 29853 raft_consensus.cc:482] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:03.349634 29853 raft_consensus.cc:504] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:03.317552 29796 tablet_service.cc:1467] Processing CreateTablet for tablet 25e07220db7444ae97f294d70eda6a93 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20240501 20:01:03.318594 29851 ts_tablet_manager.cc:1398] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.278s user 0.020s sys 0.022s
I20240501 20:01:03.326786 29644 tablet_service.cc:1467] Processing CreateTablet for tablet 3af08384057a4d85977481472f460168 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20240501 20:01:03.358772 29644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3af08384057a4d85977481472f460168. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.339408 29852 tablet_bootstrap.cc:492] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:03.360536 29852 ts_tablet_manager.cc:1367] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.444s user 0.013s sys 0.022s
I20240501 20:01:03.369774 29852 raft_consensus.cc:348] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:03.370756 29852 raft_consensus.cc:374] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:03.371129 29852 raft_consensus.cc:729] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:03.373510 29648 tablet_service.cc:1467] Processing CreateTablet for tablet 24a2bba62f434d1c8489b55c8fec2187 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20240501 20:01:03.373796 29853 leader_election.cc:288] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:03.374840 29648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24a2bba62f434d1c8489b55c8fec2187. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.385501 29796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25e07220db7444ae97f294d70eda6a93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.385830 29851 tablet_bootstrap.cc:492] T add9d7579351483fb268f89a44e95d8a P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:03.393873 29716 tablet_service.cc:1467] Processing CreateTablet for tablet 77be7101c0fe4085b6706296f609870a (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20240501 20:01:03.395099 29716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77be7101c0fe4085b6706296f609870a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.398370 29725 tablet_service.cc:1467] Processing CreateTablet for tablet a8e9f75387564275a60d38e9a910319c (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20240501 20:01:03.399650 29725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8e9f75387564275a60d38e9a910319c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.400408 29851 tablet_bootstrap.cc:654] T add9d7579351483fb268f89a44e95d8a P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:03.371937 29852 consensus_queue.cc:260] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:03.413142 29795 tablet_service.cc:1467] Processing CreateTablet for tablet ece8d4ed84c34b748f7b6272b31c90ac (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20240501 20:01:03.414637 29795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ece8d4ed84c34b748f7b6272b31c90ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.385051 29799 tablet_service.cc:1467] Processing CreateTablet for tablet 3c5345f6eb62449ba69b4a9c2e349b31 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20240501 20:01:03.417354 29642 tablet_service.cc:1467] Processing CreateTablet for tablet bbcf30cf66714efdac997862dafbe4a2 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20240501 20:01:03.379249 29792 tablet_service.cc:1467] Processing CreateTablet for tablet a5525fc4b82a44b0991c3dd6098fde81 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20240501 20:01:03.436692 29723 tablet_service.cc:1467] Processing CreateTablet for tablet b337b8e3e38d4ad6b1076fbd0d0a1609 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20240501 20:01:03.440881 29724 tablet_service.cc:1467] Processing CreateTablet for tablet 25e07220db7444ae97f294d70eda6a93 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20240501 20:01:03.444845 29723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b337b8e3e38d4ad6b1076fbd0d0a1609. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.445561 29717 tablet_service.cc:1467] Processing CreateTablet for tablet 211e27a3b0084b89a6dd7550849b5a65 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20240501 20:01:03.446483 29642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbcf30cf66714efdac997862dafbe4a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.447275 29646 tablet_service.cc:1467] Processing CreateTablet for tablet 0fb27193abbf4644a48662909bc7b04a (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20240501 20:01:03.448083 29649 tablet_service.cc:1467] Processing CreateTablet for tablet a7c597f15e844f30837012baed4b0501 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20240501 20:01:03.460986 29649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7c597f15e844f30837012baed4b0501. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.461418 29720 tablet_service.cc:1467] Processing CreateTablet for tablet d48fd0a8e289401aa7417c42ff2c81b2 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20240501 20:01:03.450639 29721 tablet_service.cc:1467] Processing CreateTablet for tablet c651044a305c4168805825a53d910899 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20240501 20:01:03.464792 29799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c5345f6eb62449ba69b4a9c2e349b31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.470950 29792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5525fc4b82a44b0991c3dd6098fde81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.468716 29717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 211e27a3b0084b89a6dd7550849b5a65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.473786 29644 tablet_service.cc:1467] Processing CreateTablet for tablet 458740ab5c444b6ab3d1991c69db5084 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20240501 20:01:03.477663 29849 tablet_bootstrap.cc:492] T f6d88139f89c47ff84cad674147aae0f P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:03.474617 29721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c651044a305c4168805825a53d910899. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.478241 29849 ts_tablet_manager.cc:1367] T f6d88139f89c47ff84cad674147aae0f P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.459s user 0.041s sys 0.056s
I20240501 20:01:03.480362 29720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d48fd0a8e289401aa7417c42ff2c81b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.485618 29718 tablet_service.cc:1467] Processing CreateTablet for tablet 5525f546426c47a38edf7f56ad04b7ab (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20240501 20:01:03.486949 29718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5525f546426c47a38edf7f56ad04b7ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.475029 29641 tablet_service.cc:1467] Processing CreateTablet for tablet 07bdb8b09b8c4ef4b8e49daa9b8c01d6 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20240501 20:01:03.476755 29646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fb27193abbf4644a48662909bc7b04a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.490765 29793 tablet_service.cc:1467] Processing CreateTablet for tablet 36057b3f24d44b9aac3a1378053a395f (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20240501 20:01:03.455772 29863 raft_consensus.cc:482] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:03.477241 29790 tablet_service.cc:1467] Processing CreateTablet for tablet 5525f546426c47a38edf7f56ad04b7ab (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20240501 20:01:03.492368 29724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25e07220db7444ae97f294d70eda6a93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.492514 29644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 458740ab5c444b6ab3d1991c69db5084. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.504155 29863 raft_consensus.cc:504] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:03.508419 29641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07bdb8b09b8c4ef4b8e49daa9b8c01d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.492012 29793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36057b3f24d44b9aac3a1378053a395f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.519810 29791 tablet_service.cc:1467] Processing CreateTablet for tablet 03f5d1686214485d91a0baf4b870740e (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20240501 20:01:03.494127 29647 tablet_service.cc:1467] Processing CreateTablet for tablet 010ceabf3e5a4b1c9330b80dade765bb (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20240501 20:01:03.525068 29790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5525f546426c47a38edf7f56ad04b7ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.535022 29851 tablet_bootstrap.cc:492] T add9d7579351483fb268f89a44e95d8a P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:03.535552 29851 ts_tablet_manager.cc:1367] T add9d7579351483fb268f89a44e95d8a P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.150s user 0.011s sys 0.005s
I20240501 20:01:03.499459 29798 tablet_service.cc:1467] Processing CreateTablet for tablet 0374bb790ed345ed90452595f71383a2 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20240501 20:01:03.536676 29716 tablet_service.cc:1467] Processing CreateTablet for tablet a5525fc4b82a44b0991c3dd6098fde81 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20240501 20:01:03.536940 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05df43bdfa479b902b1da297fb5974" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
W20240501 20:01:03.543188 29773 leader_election.cc:341] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:03.550895 29647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 010ceabf3e5a4b1c9330b80dade765bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.555049 29851 raft_consensus.cc:348] T add9d7579351483fb268f89a44e95d8a P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:03.556106 29851 raft_consensus.cc:374] T add9d7579351483fb268f89a44e95d8a P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:03.556480 29851 raft_consensus.cc:729] T add9d7579351483fb268f89a44e95d8a P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:03.512627 29849 raft_consensus.cc:348] T f6d88139f89c47ff84cad674147aae0f P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:03.520167 29796 tablet_service.cc:1467] Processing CreateTablet for tablet 24a2bba62f434d1c8489b55c8fec2187 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20240501 20:01:03.544418 29716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5525fc4b82a44b0991c3dd6098fde81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.580901 29797 tablet_service.cc:1467] Processing CreateTablet for tablet 6797d82dda2948d8a7445fcd33e44752 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20240501 20:01:03.581530 29722 tablet_service.cc:1467] Processing CreateTablet for tablet 3c5345f6eb62449ba69b4a9c2e349b31 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20240501 20:01:03.592340 29851 consensus_queue.cc:260] T add9d7579351483fb268f89a44e95d8a P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:03.594614 29643 tablet_service.cc:1467] Processing CreateTablet for tablet 77be7101c0fe4085b6706296f609870a (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20240501 20:01:03.595520 29851 ts_tablet_manager.cc:1398] T add9d7579351483fb268f89a44e95d8a P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.060s user 0.004s sys 0.004s
I20240501 20:01:03.604454 29722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c5345f6eb62449ba69b4a9c2e349b31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.612862 29856 raft_consensus.cc:482] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:03.613371 29856 raft_consensus.cc:504] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } }
I20240501 20:01:03.545099 29863 leader_election.cc:288] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:03.624153 29852 ts_tablet_manager.cc:1398] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.263s user 0.010s sys 0.004s
I20240501 20:01:03.564615 29849 raft_consensus.cc:374] T f6d88139f89c47ff84cad674147aae0f P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:03.625039 29643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77be7101c0fe4085b6706296f609870a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.629112 29649 tablet_service.cc:1467] Processing CreateTablet for tablet 25e07220db7444ae97f294d70eda6a93 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20240501 20:01:03.637604 29851 tablet_bootstrap.cc:492] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:03.631747 29791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03f5d1686214485d91a0baf4b870740e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.644081 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05df43bdfa479b902b1da297fb5974" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:03.629767 29856 leader_election.cc:288] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861), 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959)
I20240501 20:01:03.630172 29849 raft_consensus.cc:729] T f6d88139f89c47ff84cad674147aae0f P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:03.628603 29795 tablet_service.cc:1467] Processing CreateTablet for tablet aa8cd40cb4044d18a86001ce6ddf97ab (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20240501 20:01:03.652873 29649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25e07220db7444ae97f294d70eda6a93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.655009 29650 tablet_service.cc:1467] Processing CreateTablet for tablet b337b8e3e38d4ad6b1076fbd0d0a1609 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20240501 20:01:03.659685 29650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b337b8e3e38d4ad6b1076fbd0d0a1609. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.676177 29794 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41856 (request call id 36) took 1037 ms. Trace:
I20240501 20:01:03.676602 29849 consensus_queue.cc:260] T f6d88139f89c47ff84cad674147aae0f P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:03.678458 29794 rpcz_store.cc:276] 0501 20:01:02.634517 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.313242 (+678725us) service_pool.cc:225] Handling call
0501 20:01:03.315211 (+ 1969us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:03.315302 (+ 91us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:03.485093 (+169791us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:03.635386 (+150293us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:03.672301 (+ 36915us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:03.681133 29794 tablet_service.cc:1467] Processing CreateTablet for tablet ec073fe645614fe28e7837de3c22510b (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20240501 20:01:03.625805 29723 tablet_service.cc:1467] Processing CreateTablet for tablet ece8d4ed84c34b748f7b6272b31c90ac (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20240501 20:01:03.683487 29723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ece8d4ed84c34b748f7b6272b31c90ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.688690 29852 tablet_bootstrap.cc:492] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:03.632612 29645 tablet_service.cc:1467] Processing CreateTablet for tablet a8e9f75387564275a60d38e9a910319c (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20240501 20:01:03.700990 29645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8e9f75387564275a60d38e9a910319c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.681464 29849 ts_tablet_manager.cc:1398] T f6d88139f89c47ff84cad674147aae0f P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.203s user 0.007s sys 0.003s
W20240501 20:01:03.708199 29772 leader_election.cc:341] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:03.708755 29772 leader_election.cc:302] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:03.709668 29853 raft_consensus.cc:2726] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:03.709918 29724 tablet_service.cc:1467] Processing CreateTablet for tablet 36057b3f24d44b9aac3a1378053a395f (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
W20240501 20:01:03.710996 29772 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 57960.2us
I20240501 20:01:03.711580 29724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36057b3f24d44b9aac3a1378053a395f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.714622 29796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24a2bba62f434d1c8489b55c8fec2187. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.715186 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfac893b4b4f457694a4eb184b6abb53" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:03.732337 29797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6797d82dda2948d8a7445fcd33e44752. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.738519 29794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec073fe645614fe28e7837de3c22510b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.741840 29851 tablet_bootstrap.cc:654] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:03.718371 29719 tablet_service.cc:1467] Processing CreateTablet for tablet 24a2bba62f434d1c8489b55c8fec2187 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20240501 20:01:03.720499 29720 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:36258 (request call id 36) took 1025 ms. Trace:
I20240501 20:01:03.721310 29792 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41856 (request call id 37) took 1086 ms. Trace:
I20240501 20:01:03.748461 29720 rpcz_store.cc:276] 0501 20:01:02.695269 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.460376 (+765107us) service_pool.cc:225] Handling call
0501 20:01:03.477051 (+ 16675us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:03.477143 (+ 92us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:03.532011 (+ 54868us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:03.717435 (+185424us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:03.720414 (+ 2979us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1238400}
I20240501 20:01:03.751195 29720 tablet_service.cc:1467] Processing CreateTablet for tablet 6797d82dda2948d8a7445fcd33e44752 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20240501 20:01:03.725762 29642 tablet_service.cc:1467] Processing CreateTablet for tablet 211e27a3b0084b89a6dd7550849b5a65 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20240501 20:01:03.728734 29849 tablet_bootstrap.cc:492] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:03.748528 29792 rpcz_store.cc:276] 0501 20:01:02.634887 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.378334 (+743447us) service_pool.cc:225] Handling call
0501 20:01:03.440138 (+ 61804us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:03.441424 (+ 1286us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:03.573271 (+131847us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:03.692541 (+119270us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:03.721002 (+ 28461us) spinlock_profiling.cc:243] Waited 6.09 ms on lock 0x7b58000d2c98. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe454797b51 00007fe4549193b1 00007fe4549199e2 00007fe45d8345f9 00007fe45c748268 00007fe45c80bd33 00007fe45c807b4f 00007fe45c808301 00007fe45c7b9619 00007fe455db7c64 00007fe455db7bf2
0501 20:01:03.721256 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14003968}
I20240501 20:01:03.722540 29799 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41856 (request call id 38) took 1087 ms. Trace:
I20240501 20:01:03.769690 29799 rpcz_store.cc:276] 0501 20:01:02.635292 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.384094 (+748802us) service_pool.cc:225] Handling call
0501 20:01:03.418105 (+ 34011us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:03.422415 (+ 4310us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:03.573609 (+151194us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:03.695511 (+121902us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:03.722248 (+ 26737us) spinlock_profiling.cc:243] Waited 2.56 ms on lock 0x7b58000d2c98. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe454797b51 00007fe4549193b1 00007fe4549199e2 00007fe45d8345f9 00007fe45c748268 00007fe45c80bd33 00007fe45c807b4f 00007fe45c808301 00007fe45c7b9619 00007fe455db7c64 00007fe455db7bf2
0501 20:01:03.722487 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5883136}
I20240501 20:01:03.770813 29792 tablet_service.cc:1467] Processing CreateTablet for tablet 3c1a08b455b94ae089396a15c837ddc2 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
W20240501 20:01:03.753659 29625 leader_election.cc:341] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:03.758419 29852 tablet_bootstrap.cc:654] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:03.771786 29799 tablet_service.cc:1467] Processing CreateTablet for tablet fa7e89c0572e4ee7be8e5315b459bdb3 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20240501 20:01:03.774530 29649 tablet_service.cc:1467] Processing CreateTablet for tablet a5525fc4b82a44b0991c3dd6098fde81 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20240501 20:01:03.775689 29649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5525fc4b82a44b0991c3dd6098fde81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.760214 29641 tablet_service.cc:1467] Processing CreateTablet for tablet c651044a305c4168805825a53d910899 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20240501 20:01:03.760818 29720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6797d82dda2948d8a7445fcd33e44752. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.761159 29795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa8cd40cb4044d18a86001ce6ddf97ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.778151 29792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c1a08b455b94ae089396a15c837ddc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.778867 29799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa7e89c0572e4ee7be8e5315b459bdb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.780321 29648 tablet_service.cc:1467] Processing CreateTablet for tablet 3c5345f6eb62449ba69b4a9c2e349b31 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20240501 20:01:03.778898 29719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24a2bba62f434d1c8489b55c8fec2187. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.775292 29725 tablet_service.cc:1467] Processing CreateTablet for tablet 0374bb790ed345ed90452595f71383a2 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20240501 20:01:03.765208 29790 tablet_service.cc:1467] Processing CreateTablet for tablet 8edcd903224d48f889b46e1b5318e0ec (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20240501 20:01:03.782598 29798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0374bb790ed345ed90452595f71383a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.781649 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfac893b4b4f457694a4eb184b6abb53" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
W20240501 20:01:03.787652 29624 leader_election.cc:341] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:03.788105 29624 leader_election.cc:302] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:03.789176 29863 raft_consensus.cc:2726] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:03.785616 29646 tablet_service.cc:1467] Processing CreateTablet for tablet d48fd0a8e289401aa7417c42ff2c81b2 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20240501 20:01:03.796595 29641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c651044a305c4168805825a53d910899. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.798483 29648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c5345f6eb62449ba69b4a9c2e349b31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.803397 29790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8edcd903224d48f889b46e1b5318e0ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.801746 29646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d48fd0a8e289401aa7417c42ff2c81b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.816738 29642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 211e27a3b0084b89a6dd7550849b5a65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.822556 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69915dd5992641e3a24f7f45f1cd5c0c" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:03.824155 29700 leader_election.cc:341] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:03.828451 29717 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:36258 (request call id 34) took 1134 ms. Trace:
I20240501 20:01:03.829185 29793 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41856 (request call id 42) took 1066 ms. Trace:
I20240501 20:01:03.828898 29717 rpcz_store.cc:276] 0501 20:01:02.694242 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.442287 (+748045us) service_pool.cc:225] Handling call
0501 20:01:03.468090 (+ 25803us) spinlock_profiling.cc:243] Waited 2.54 ms on lock 0x7b58001b3098. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe454797b51 00007fe4549193b1 00007fe454919262 00007fe45d80b889 00007fe45c7483c0 00007fe45c82a791 00007fe45c807f03 00007fe45c7b9619 00007fe455db7c64 00007fe455db7bf2 00007fe455db7b21
0501 20:01:03.468226 (+ 136us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:03.468320 (+ 94us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:03.559168 (+ 90848us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:03.789043 (+229875us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:03.828386 (+ 39343us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5852928}
I20240501 20:01:03.829746 29793 rpcz_store.cc:276] 0501 20:01:02.763157 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.489835 (+726678us) service_pool.cc:225] Handling call
0501 20:01:03.491566 (+ 1731us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:03.491671 (+ 105us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:03.753947 (+262276us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:03.828664 (+ 74717us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:03.829126 (+ 462us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:03.828361 29721 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:36258 (request call id 35) took 1133 ms. Trace:
I20240501 20:01:03.831959 29717 tablet_service.cc:1467] Processing CreateTablet for tablet aa8cd40cb4044d18a86001ce6ddf97ab (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20240501 20:01:03.831640 29721 rpcz_store.cc:276] 0501 20:01:02.694946 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.449779 (+754833us) service_pool.cc:225] Handling call
0501 20:01:03.469148 (+ 19369us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:03.469236 (+ 88us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:03.585056 (+115820us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:03.727202 (+142146us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:03.828291 (+101089us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:03.833695 29717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa8cd40cb4044d18a86001ce6ddf97ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.833410 29647 tablet_service.cc:1467] Processing CreateTablet for tablet ece8d4ed84c34b748f7b6272b31c90ac (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20240501 20:01:03.834792 29647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ece8d4ed84c34b748f7b6272b31c90ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.821132 29849 tablet_bootstrap.cc:654] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:03.835449 29721 tablet_service.cc:1467] Processing CreateTablet for tablet ec073fe645614fe28e7837de3c22510b (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20240501 20:01:03.836649 29721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec073fe645614fe28e7837de3c22510b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.841684 29722 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:36258 (request call id 38) took 1145 ms. Trace:
I20240501 20:01:03.842141 29722 rpcz_store.cc:276] 0501 20:01:02.695957 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.580192 (+884235us) service_pool.cc:225] Handling call
0501 20:01:03.582715 (+ 2523us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:03.582855 (+ 140us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:03.671803 (+ 88948us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:03.810347 (+138544us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:03.840256 (+ 29909us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:03.843794 29718 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:36258 (request call id 40) took 1147 ms. Trace:
I20240501 20:01:03.844108 29722 tablet_service.cc:1467] Processing CreateTablet for tablet 03f5d1686214485d91a0baf4b870740e (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20240501 20:01:03.845283 29725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0374bb790ed345ed90452595f71383a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.846024 29722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03f5d1686214485d91a0baf4b870740e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.850586 29853 raft_consensus.cc:482] T f6d88139f89c47ff84cad674147aae0f P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:03.851070 29853 raft_consensus.cc:504] T f6d88139f89c47ff84cad674147aae0f P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:03.853070 29853 leader_election.cc:288] T f6d88139f89c47ff84cad674147aae0f P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:03.856303 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d88139f89c47ff84cad674147aae0f" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:03.856427 29650 tablet_service.cc:1467] Processing CreateTablet for tablet 5525f546426c47a38edf7f56ad04b7ab (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
W20240501 20:01:03.857890 29772 leader_election.cc:341] T f6d88139f89c47ff84cad674147aae0f P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:03.853744 29718 rpcz_store.cc:276] 0501 20:01:02.696572 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.484522 (+787950us) service_pool.cc:225] Handling call
0501 20:01:03.486501 (+ 1979us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:03.486591 (+ 90us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:03.728734 (+242143us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:03.815213 (+ 86479us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:03.843738 (+ 28525us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:03.867408 29718 tablet_service.cc:1467] Processing CreateTablet for tablet 8edcd903224d48f889b46e1b5318e0ec (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20240501 20:01:03.853543 29723 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:36258 (request call id 39) took 1157 ms. Trace:
I20240501 20:01:03.868324 29723 rpcz_store.cc:276] 0501 20:01:02.696280 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.611184 (+914904us) service_pool.cc:225] Handling call
0501 20:01:03.682998 (+ 71814us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:03.683093 (+ 95us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:03.745669 (+ 62576us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:03.853077 (+107408us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:03.853489 (+ 412us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:03.833202 29793 tablet_service.cc:1467] Processing CreateTablet for tablet 7be5f38dc96d4fad80d83eea0822aa22 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20240501 20:01:03.857872 29650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5525f546426c47a38edf7f56ad04b7ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.876542 29716 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:36258 (request call id 37) took 1180 ms. Trace:
I20240501 20:01:03.881788 29852 tablet_bootstrap.cc:492] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:03.882364 29852 ts_tablet_manager.cc:1367] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.226s user 0.011s sys 0.017s
I20240501 20:01:03.881855 29716 rpcz_store.cc:276] 0501 20:01:02.695644 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.510831 (+815187us) service_pool.cc:225] Handling call
0501 20:01:03.537525 (+ 26694us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:03.537650 (+ 125us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:03.745860 (+208210us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:03.875996 (+130136us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:03.876483 (+ 487us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:03.884431 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69915dd5992641e3a24f7f45f1cd5c0c" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:03.885231 29852 raft_consensus.cc:348] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } }
I20240501 20:01:03.885906 29852 raft_consensus.cc:374] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:03.886198 29852 raft_consensus.cc:729] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:03.886411 29796 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41856 (request call id 43) took 1098 ms. Trace:
W20240501 20:01:03.887027 29698 leader_election.cc:341] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:03.886884 29852 consensus_queue.cc:260] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } }
I20240501 20:01:03.887553 29698 leader_election.cc:302] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:03.889001 29856 raft_consensus.cc:2726] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:03.889328 29648 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59124 (request call id 38) took 1021 ms. Trace:
I20240501 20:01:03.889660 29648 rpcz_store.cc:276] 0501 20:01:02.867410 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.779395 (+911985us) service_pool.cc:225] Handling call
0501 20:01:03.781228 (+ 1833us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:03.781333 (+ 105us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:03.848395 (+ 67062us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:03.888848 (+ 40453us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:03.889275 (+ 427us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":87040}
I20240501 20:01:03.883940 29716 tablet_service.cc:1467] Processing CreateTablet for tablet fa7e89c0572e4ee7be8e5315b459bdb3 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20240501 20:01:03.891739 29852 ts_tablet_manager.cc:1398] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20240501 20:01:03.872007 29791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41856 (request call id 47) took 1055 ms. Trace:
I20240501 20:01:03.859894 29797 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41856 (request call id 44) took 1043 ms. Trace:
I20240501 20:01:03.892356 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d88139f89c47ff84cad674147aae0f" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:03.892396 29791 rpcz_store.cc:276] 0501 20:01:02.816540 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.517723 (+701183us) service_pool.cc:225] Handling call
0501 20:01:03.541392 (+ 23669us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:03.547390 (+ 5998us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:03.812916 (+265526us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:03.871492 (+ 58576us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:03.871943 (+ 451us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:03.892647 29797 rpcz_store.cc:276] 0501 20:01:02.815366 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.513730 (+698364us) service_pool.cc:225] Handling call
0501 20:01:03.666861 (+153131us) spinlock_profiling.cc:243] Waited 34.8 ms on lock 0x7b58000d2c98. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe454797b51 00007fe4549193b1 00007fe454919262 00007fe45d80b889 00007fe45c7483c0 00007fe45c82a791 00007fe45c807f03 00007fe45c7b9619 00007fe455db7c64 00007fe455db7bf2 00007fe455db7b21
0501 20:01:03.667002 (+ 141us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:03.667091 (+ 89us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:03.755058 (+ 87967us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:03.787988 (+ 32930us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:03.859098 (+ 71110us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8946,"spinlock_wait_cycles":79935232}
I20240501 20:01:03.894728 29791 tablet_service.cc:1467] Processing CreateTablet for tablet fc312e7affec4f67ada04db849694bdc (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20240501 20:01:03.895109 29797 tablet_service.cc:1467] Processing CreateTablet for tablet 7ed27033ab1645e9ab0600ab635d9589 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20240501 20:01:03.876879 29793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7be5f38dc96d4fad80d83eea0822aa22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.900250 29856 raft_consensus.cc:482] T add9d7579351483fb268f89a44e95d8a P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:03.900579 29720 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:36258 (request call id 44) took 1124 ms. Trace:
I20240501 20:01:03.900861 29856 raft_consensus.cc:504] T add9d7579351483fb268f89a44e95d8a P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:03.900975 29720 rpcz_store.cc:276] 0501 20:01:02.775769 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.750267 (+974498us) service_pool.cc:225] Handling call
0501 20:01:03.751966 (+ 1699us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:03.752557 (+ 591us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:03.826622 (+ 74065us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:03.869035 (+ 42413us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:03.900503 (+ 31468us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:03.874272 29644 tablet_service.cc:1467] Processing CreateTablet for tablet 8edcd903224d48f889b46e1b5318e0ec (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20240501 20:01:03.903749 29720 tablet_service.cc:1467] Processing CreateTablet for tablet 7be5f38dc96d4fad80d83eea0822aa22 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20240501 20:01:03.904944 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add9d7579351483fb268f89a44e95d8a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:03.906081 29856 leader_election.cc:288] T add9d7579351483fb268f89a44e95d8a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:03.907052 29700 leader_election.cc:341] T add9d7579351483fb268f89a44e95d8a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:03.877628 29718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8edcd903224d48f889b46e1b5318e0ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.908617 29716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa7e89c0572e4ee7be8e5315b459bdb3. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 20:01:03.910712 29773 leader_election.cc:341] T f6d88139f89c47ff84cad674147aae0f P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:03.911163 29773 leader_election.cc:302] T f6d88139f89c47ff84cad674147aae0f P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:03.912562 29853 raft_consensus.cc:2726] T f6d88139f89c47ff84cad674147aae0f P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:03.886839 29796 rpcz_store.cc:276] 0501 20:01:02.814465 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.512780 (+698315us) service_pool.cc:225] Handling call
0501 20:01:03.597819 (+ 85039us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:03.597963 (+ 144us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:03.801703 (+203740us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:03.885928 (+ 84225us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:03.886351 (+ 423us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:03.918079 29799 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41856 (request call id 50) took 1038 ms. Trace:
I20240501 20:01:03.918589 29799 rpcz_store.cc:276] 0501 20:01:02.879504 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.770890 (+891386us) service_pool.cc:225] Handling call
0501 20:01:03.773347 (+ 2457us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:03.773543 (+ 196us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:03.832227 (+ 58684us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:03.917452 (+ 85225us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:03.918016 (+ 564us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:03.919740 29797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ed27033ab1645e9ab0600ab635d9589. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.924401 29791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc312e7affec4f67ada04db849694bdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.857609 29642 tablet_service.cc:1467] Processing CreateTablet for tablet 6797d82dda2948d8a7445fcd33e44752 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20240501 20:01:03.927271 29721 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:36258 (request call id 46) took 1139 ms. Trace:
I20240501 20:01:03.930023 29721 rpcz_store.cc:276] 0501 20:01:02.789007 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.834546 (+1045539us) service_pool.cc:225] Handling call
0501 20:01:03.836241 (+ 1695us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:03.836329 (+ 88us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:03.891583 (+ 55254us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:03.908000 (+ 16417us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:03.927209 (+ 19209us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:03.931439 29799 tablet_service.cc:1467] Processing CreateTablet for tablet 31fff64b453a423eb0f567b3171564fb (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20240501 20:01:03.932806 29721 tablet_service.cc:1467] Processing CreateTablet for tablet fc312e7affec4f67ada04db849694bdc (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20240501 20:01:03.946062 29642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6797d82dda2948d8a7445fcd33e44752. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.931080 29720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7be5f38dc96d4fad80d83eea0822aa22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.947269 29799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31fff64b453a423eb0f567b3171564fb. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 20:01:03.949055 29606 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240501 20:01:03.949684 29792 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41856 (request call id 49) took 1073 ms. Trace:
I20240501 20:01:03.950067 29792 rpcz_store.cc:276] 0501 20:01:02.876437 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.769857 (+893420us) service_pool.cc:225] Handling call
0501 20:01:03.771611 (+ 1754us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:03.771702 (+ 91us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:03.873375 (+101673us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:03.899960 (+ 26585us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:03.949625 (+ 49665us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18459}
I20240501 20:01:03.952594 29721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc312e7affec4f67ada04db849694bdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.952018 29792 tablet_service.cc:1467] Processing CreateTablet for tablet 77a35893b91247a99912eba787d787cf (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20240501 20:01:03.954490 29792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77a35893b91247a99912eba787d787cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.956722 29644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8edcd903224d48f889b46e1b5318e0ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.958482 29851 tablet_bootstrap.cc:492] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:03.958945 29851 ts_tablet_manager.cc:1367] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.322s user 0.017s sys 0.018s
I20240501 20:01:03.959869 29793 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41856 (request call id 51) took 1069 ms. Trace:
I20240501 20:01:03.960285 29793 rpcz_store.cc:276] 0501 20:01:02.890570 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.832174 (+941604us) service_pool.cc:225] Handling call
0501 20:01:03.876399 (+ 44225us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:03.876495 (+ 96us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:03.936479 (+ 59984us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:03.959271 (+ 22792us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:03.959791 (+ 520us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:03.934228 29719 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:36258 (request call id 43) took 1165 ms. Trace:
I20240501 20:01:03.925654 29796 tablet_service.cc:1467] Processing CreateTablet for tablet a8787bc4d6b44649b9138d906730110a (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20240501 20:01:03.975091 29645 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59124 (request call id 32) took 1116 ms. Trace:
I20240501 20:01:03.975791 29796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8787bc4d6b44649b9138d906730110a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.975457 29645 rpcz_store.cc:276] 0501 20:01:02.859151 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.611185 (+752034us) service_pool.cc:225] Handling call
0501 20:01:03.700549 (+ 89364us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:03.700641 (+ 92us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:03.865501 (+164860us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:03.920279 (+ 54778us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:03.975028 (+ 54749us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18602}
I20240501 20:01:03.977561 29645 tablet_service.cc:1467] Processing CreateTablet for tablet aa8cd40cb4044d18a86001ce6ddf97ab (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20240501 20:01:03.978750 29645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa8cd40cb4044d18a86001ce6ddf97ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.878954 29723 tablet_service.cc:1467] Processing CreateTablet for tablet 3c1a08b455b94ae089396a15c837ddc2 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20240501 20:01:03.980870 29723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c1a08b455b94ae089396a15c837ddc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.936676 29849 tablet_bootstrap.cc:492] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:03.944916 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add9d7579351483fb268f89a44e95d8a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:03.984674 29849 ts_tablet_manager.cc:1367] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.256s user 0.009s sys 0.011s
W20240501 20:01:03.985487 29698 leader_election.cc:341] T add9d7579351483fb268f89a44e95d8a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:03.985921 29698 leader_election.cc:302] T add9d7579351483fb268f89a44e95d8a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:03.986937 29856 raft_consensus.cc:2726] T add9d7579351483fb268f89a44e95d8a P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:03.987383 29849 raft_consensus.cc:348] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:03.944976 29648 tablet_service.cc:1467] Processing CreateTablet for tablet 36057b3f24d44b9aac3a1378053a395f (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20240501 20:01:03.989326 29648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36057b3f24d44b9aac3a1378053a395f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.991286 29718 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:36258 (request call id 48) took 1176 ms. Trace:
I20240501 20:01:03.991640 29717 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:36258 (request call id 45) took 1210 ms. Trace:
I20240501 20:01:03.992277 29849 raft_consensus.cc:374] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:03.993778 29646 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59124 (request call id 36) took 1127 ms. Trace:
I20240501 20:01:03.996570 29849 raft_consensus.cc:729] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:03.996757 29646 rpcz_store.cc:276] 0501 20:01:02.866788 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.767700 (+900912us) service_pool.cc:225] Handling call
0501 20:01:03.797299 (+ 29599us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:03.797843 (+ 544us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:03.849483 (+ 51640us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:03.970839 (+121356us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:03.993699 (+ 22860us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22334}
I20240501 20:01:03.998819 29792 tablet_service.cc:1467] Processing CreateTablet for tablet 4e49dd5180f04313ab33942fb3ddc147 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20240501 20:01:04.000588 29792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e49dd5180f04313ab33942fb3ddc147. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.992746 29641 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59124 (request call id 35) took 1132 ms. Trace:
I20240501 20:01:04.002220 29641 rpcz_store.cc:276] 0501 20:01:02.862348 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.759275 (+896927us) service_pool.cc:225] Handling call
0501 20:01:03.796116 (+ 36841us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:03.796227 (+ 111us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:03.867667 (+ 71440us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:03.992231 (+124564us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:03.992679 (+ 448us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":64128}
I20240501 20:01:04.003720 29794 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41856 (request call id 46) took 1187 ms. Trace:
I20240501 20:01:04.004936 29794 rpcz_store.cc:276] 0501 20:01:02.816175 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.680170 (+863995us) service_pool.cc:225] Handling call
0501 20:01:03.715970 (+ 35800us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:03.716130 (+ 160us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:03.877891 (+161761us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:04.001930 (+124039us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:04.003136 (+ 1206us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19993,"spinlock_wait_cycles":2944}
I20240501 20:01:03.971110 29716 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:36258 (request call id 50) took 1090 ms. Trace:
I20240501 20:01:03.972769 29725 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:36258 (request call id 41) took 1275 ms. Trace:
I20240501 20:01:03.997495 29791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41856 (request call id 52) took 1091 ms. Trace:
I20240501 20:01:04.009518 29725 rpcz_store.cc:276] 0501 20:01:02.696872 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.726819 (+1029947us) service_pool.cc:225] Handling call
0501 20:01:03.818237 (+ 91418us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:03.820568 (+ 2331us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:03.922910 (+102342us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:03.972258 (+ 49348us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:03.972711 (+ 453us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:04.012887 29725 tablet_service.cc:1467] Processing CreateTablet for tablet 7ed27033ab1645e9ab0600ab635d9589 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20240501 20:01:04.014569 29725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ed27033ab1645e9ab0600ab635d9589. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.992022 29717 rpcz_store.cc:276] 0501 20:01:02.780962 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.830959 (+1049997us) service_pool.cc:225] Handling call
0501 20:01:03.832890 (+ 1931us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:03.833296 (+ 406us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:03.968817 (+135521us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:03.991198 (+ 22381us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:03.991582 (+ 384us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:03.973441 29647 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59124 (request call id 39) took 1105 ms. Trace:
I20240501 20:01:04.030036 29647 rpcz_store.cc:276] 0501 20:01:02.867688 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.832463 (+964775us) service_pool.cc:225] Handling call
0501 20:01:03.834330 (+ 1867us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:03.834416 (+ 86us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:03.886041 (+ 51625us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:03.911279 (+ 25238us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:03.973387 (+ 62108us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26601}
I20240501 20:01:04.032007 29647 tablet_service.cc:1467] Processing CreateTablet for tablet fa7e89c0572e4ee7be8e5315b459bdb3 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20240501 20:01:04.032356 29851 raft_consensus.cc:348] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.033138 29851 raft_consensus.cc:374] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:03.973944 29643 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59124 (request call id 31) took 1115 ms. Trace:
I20240501 20:01:04.033437 29851 raft_consensus.cc:729] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:04.033710 29643 rpcz_store.cc:276] 0501 20:01:02.858845 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.593172 (+734327us) service_pool.cc:225] Handling call
0501 20:01:03.595962 (+ 2790us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:03.596106 (+ 144us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:03.761507 (+165401us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:03.888799 (+127292us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:03.973891 (+ 85092us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17487}
I20240501 20:01:04.035073 29790 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41856 (request call id 48) took 1218 ms. Trace:
I20240501 20:01:04.035807 29643 tablet_service.cc:1467] Processing CreateTablet for tablet 7be5f38dc96d4fad80d83eea0822aa22 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20240501 20:01:04.033268 29647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa7e89c0572e4ee7be8e5315b459bdb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:04.037604 29795 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41856 (request call id 45) took 1221 ms. Trace:
I20240501 20:01:03.974640 29720 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:36258 (request call id 51) took 1085 ms. Trace:
I20240501 20:01:04.038262 29643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7be5f38dc96d4fad80d83eea0822aa22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:04.038437 29795 rpcz_store.cc:276] 0501 20:01:02.815768 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.601838 (+786070us) service_pool.cc:225] Handling call
0501 20:01:03.677798 (+ 75960us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:03.715679 (+ 37881us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:03.898304 (+182625us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:04.037078 (+138774us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:04.037547 (+ 469us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:03.945179 29649 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59124 (request call id 37) took 1078 ms. Trace:
I20240501 20:01:04.043300 29649 rpcz_store.cc:276] 0501 20:01:02.867130 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.773674 (+906544us) service_pool.cc:225] Handling call
0501 20:01:03.775266 (+ 1592us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:03.775355 (+ 89us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:03.813997 (+ 38642us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:03.898244 (+ 84247us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:03.945123 (+ 46879us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:04.045250 29649 tablet_service.cc:1467] Processing CreateTablet for tablet fc312e7affec4f67ada04db849694bdc (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20240501 20:01:04.046476 29649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc312e7affec4f67ada04db849694bdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.962325 29793 tablet_service.cc:1467] Processing CreateTablet for tablet 84fa28fb55f44b61ba1a88ed775a78ba (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20240501 20:01:04.049182 29721 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:36258 (request call id 52) took 1143 ms. Trace:
I20240501 20:01:04.011443 29791 rpcz_store.cc:276] 0501 20:01:02.906202 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.893742 (+987540us) service_pool.cc:225] Handling call
0501 20:01:03.895548 (+ 1806us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:03.895641 (+ 93us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:03.936426 (+ 40785us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:03.996921 (+ 60495us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:03.997430 (+ 509us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24546}
I20240501 20:01:04.018263 29719 rpcz_store.cc:276] 0501 20:01:02.768507 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.717452 (+948945us) service_pool.cc:225] Handling call
0501 20:01:03.776090 (+ 58638us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:03.776827 (+ 737us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:03.912731 (+135904us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:03.933732 (+ 21001us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:03.934170 (+ 438us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:03.972913 29724 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:36258 (request call id 42) took 1274 ms. Trace:
I20240501 20:01:04.057179 29724 rpcz_store.cc:276] 0501 20:01:02.698830 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.708668 (+1009838us) service_pool.cc:225] Handling call
0501 20:01:03.710983 (+ 2315us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:03.711113 (+ 130us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:03.867224 (+156111us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:03.972488 (+105264us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:03.972848 (+ 360us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:04.057406 29717 tablet_service.cc:1467] Processing CreateTablet for tablet a8787bc4d6b44649b9138d906730110a (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20240501 20:01:04.057912 29719 tablet_service.cc:1467] Processing CreateTablet for tablet 31fff64b453a423eb0f567b3171564fb (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20240501 20:01:04.058852 29717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8787bc4d6b44649b9138d906730110a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:04.024240 29849 consensus_queue.cc:260] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.059175 29724 tablet_service.cc:1467] Processing CreateTablet for tablet 77a35893b91247a99912eba787d787cf (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20240501 20:01:04.009507 29716 rpcz_store.cc:276] 0501 20:01:02.880796 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.883001 (+1002205us) service_pool.cc:225] Handling call
0501 20:01:03.891094 (+ 8093us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:03.891282 (+ 188us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:03.937960 (+ 46678us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:03.946117 (+ 8157us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:03.971048 (+ 24931us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:04.061129 29849 ts_tablet_manager.cc:1398] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.076s user 0.008s sys 0.003s
I20240501 20:01:04.061223 29716 tablet_service.cc:1467] Processing CreateTablet for tablet 84fa28fb55f44b61ba1a88ed775a78ba (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20240501 20:01:04.062291 29849 tablet_bootstrap.cc:492] T 1ca3948261374cef8b1b206a43c00173 P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:04.062542 29716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84fa28fb55f44b61ba1a88ed775a78ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:04.062772 29721 rpcz_store.cc:276] 0501 20:01:02.905505 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.931880 (+1026375us) service_pool.cc:225] Handling call
0501 20:01:03.946816 (+ 14936us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:03.946931 (+ 115us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:03.968496 (+ 21565us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:04.048705 (+ 80209us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:04.049122 (+ 417us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:04.064575 29721 tablet_service.cc:1467] Processing CreateTablet for tablet 4e49dd5180f04313ab33942fb3ddc147 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20240501 20:01:04.065723 29721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e49dd5180f04313ab33942fb3ddc147. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:04.068359 29719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31fff64b453a423eb0f567b3171564fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:04.074623 29643 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59124 (request call id 51) took 1169 ms. Trace:
I20240501 20:01:03.991664 29718 rpcz_store.cc:276] 0501 20:01:02.814812 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.866463 (+1051651us) service_pool.cc:225] Handling call
0501 20:01:03.877164 (+ 10701us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:03.877248 (+ 84us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:03.984743 (+107495us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:03.990818 (+ 6075us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:03.991229 (+ 411us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:04.028993 29799 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41856 (request call id 55) took 1086 ms. Trace:
I20240501 20:01:04.095460 29643 rpcz_store.cc:276] 0501 20:01:02.904852 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:04.034859 (+1130007us) service_pool.cc:225] Handling call
0501 20:01:04.036761 (+ 1902us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:04.036950 (+ 189us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:04.061929 (+ 24979us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:04.074106 (+ 12177us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:04.074536 (+ 430us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:04.095974 29799 rpcz_store.cc:276] 0501 20:01:02.942179 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.930460 (+988281us) service_pool.cc:225] Handling call
0501 20:01:03.946736 (+ 16276us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:03.946862 (+ 126us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:03.955131 (+ 8269us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:04.028477 (+ 73346us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:04.028932 (+ 455us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:04.035449 29790 rpcz_store.cc:276] 0501 20:01:02.816914 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.764229 (+947315us) service_pool.cc:225] Handling call
0501 20:01:03.786390 (+ 22161us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:03.786477 (+ 87us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:03.870136 (+ 83659us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:04.034521 (+164385us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:04.035005 (+ 484us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:04.038523 29720 rpcz_store.cc:276] 0501 20:01:02.889685 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.902831 (+1013146us) service_pool.cc:225] Handling call
0501 20:01:03.928080 (+ 25249us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:03.929917 (+ 1837us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:03.963009 (+ 33092us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:03.974122 (+ 11113us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:03.974586 (+ 464us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:04.097723 29643 tablet_service.cc:1467] Processing CreateTablet for tablet 7ed27033ab1645e9ab0600ab635d9589 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20240501 20:01:04.099185 29643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ed27033ab1645e9ab0600ab635d9589. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:04.040908 29650 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59124 (request call id 40) took 1172 ms. Trace:
I20240501 20:01:03.988003 29722 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:36258 (request call id 47) took 1193 ms. Trace:
I20240501 20:01:04.100162 29722 rpcz_store.cc:276] 0501 20:01:02.794516 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.843111 (+1048595us) service_pool.cc:225] Handling call
0501 20:01:03.845361 (+ 2250us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:03.845505 (+ 144us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:03.908154 (+ 62649us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:03.975996 (+ 67842us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:03.987945 (+ 11949us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:03.974588 29852 tablet_bootstrap.cc:492] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:04.102463 29717 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:36258 (request call id 54) took 1180 ms. Trace:
I20240501 20:01:04.102823 29717 rpcz_store.cc:276] 0501 20:01:02.921991 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:04.056364 (+1134373us) service_pool.cc:225] Handling call
0501 20:01:04.058397 (+ 2033us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:04.058492 (+ 95us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:04.087383 (+ 28891us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:04.101963 (+ 14580us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:04.102404 (+ 441us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:04.049582 29793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84fa28fb55f44b61ba1a88ed775a78ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:03.967593 29798 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41856 (request call id 41) took 1204 ms. Trace:
I20240501 20:01:04.108311 29849 tablet_bootstrap.cc:654] T 1ca3948261374cef8b1b206a43c00173 P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:04.108206 29798 rpcz_store.cc:276] 0501 20:01:02.762835 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.488783 (+725948us) service_pool.cc:225] Handling call
0501 20:01:03.592755 (+103972us) spinlock_profiling.cc:243] Waited 5.25 ms on lock 0x7b58000d2c98. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe454797b51 00007fe4549193b1 00007fe454919262 00007fe45d80b889 00007fe45c7483c0 00007fe45c82a791 00007fe45c807f03 00007fe45c7b9619 00007fe455db7c64 00007fe455db7bf2 00007fe455db7b21
0501 20:01:03.597625 (+ 4870us) spinlock_profiling.cc:243] Waited 3.37 ms on lock 0x7b58000d2c98. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe454797b51 00007fe4549196dc 00007fe45491929b 00007fe45d80b889 00007fe45c7483c0 00007fe45c82a791 00007fe45c807f03 00007fe45c7b9619 00007fe455db7c64 00007fe455db7bf2 00007fe455db7b21
0501 20:01:03.600376 (+ 2751us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:03.600520 (+ 144us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:03.869956 (+269436us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:03.966380 (+ 96424us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:03.967519 (+ 1139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":19814016}
I20240501 20:01:04.053181 29851 consensus_queue.cc:260] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.110956 29851 ts_tablet_manager.cc:1398] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.152s user 0.008s sys 0.003s
I20240501 20:01:04.058758 29644 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59124 (request call id 48) took 1188 ms. Trace:
I20240501 20:01:04.075269 29796 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41856 (request call id 54) took 1150 ms. Trace:
I20240501 20:01:04.027096 29646 tablet_service.cc:1467] Processing CreateTablet for tablet 03f5d1686214485d91a0baf4b870740e (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20240501 20:01:04.112185 29851 tablet_bootstrap.cc:492] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:04.111413 29796 rpcz_store.cc:276] 0501 20:01:02.924874 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.924210 (+999336us) service_pool.cc:225] Handling call
0501 20:01:03.975320 (+ 51110us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:03.975409 (+ 89us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:04.034026 (+ 58617us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:04.074794 (+ 40768us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:04.075207 (+ 413us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:04.076380 29724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77a35893b91247a99912eba787d787cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:04.112915 29646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03f5d1686214485d91a0baf4b870740e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:04.014926 29641 tablet_service.cc:1467] Processing CreateTablet for tablet ec073fe645614fe28e7837de3c22510b (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20240501 20:01:04.078608 29797 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41856 (request call id 53) took 1154 ms. Trace:
I20240501 20:01:04.082878 29647 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59124 (request call id 50) took 1178 ms. Trace:
I20240501 20:01:04.083205 29645 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59124 (request call id 45) took 1213 ms. Trace:
I20240501 20:01:04.089210 29649 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59124 (request call id 52) took 1184 ms. Trace:
I20240501 20:01:04.116329 29647 rpcz_store.cc:276] 0501 20:01:02.904537 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:04.031080 (+1126543us) service_pool.cc:225] Handling call
0501 20:01:04.032815 (+ 1735us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:04.032901 (+ 86us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:04.049582 (+ 16681us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:04.082283 (+ 32701us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:04.082813 (+ 530us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:04.117184 29645 rpcz_store.cc:276] 0501 20:01:02.869433 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.976631 (+1107198us) service_pool.cc:225] Handling call
0501 20:01:03.978319 (+ 1688us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:03.978414 (+ 95us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:04.072361 (+ 93947us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:04.082714 (+ 10353us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:04.083145 (+ 431us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:04.117269 29649 rpcz_store.cc:276] 0501 20:01:02.905209 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:04.044271 (+1139062us) service_pool.cc:225] Handling call
0501 20:01:04.046018 (+ 1747us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:04.046125 (+ 107us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:04.069745 (+ 23620us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:04.088411 (+ 18666us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:04.089083 (+ 672us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:04.118626 29647 tablet_service.cc:1467] Processing CreateTablet for tablet a8787bc4d6b44649b9138d906730110a (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20240501 20:01:04.119459 29645 tablet_service.cc:1467] Processing CreateTablet for tablet 31fff64b453a423eb0f567b3171564fb (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20240501 20:01:04.119511 29649 tablet_service.cc:1467] Processing CreateTablet for tablet 77a35893b91247a99912eba787d787cf (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20240501 20:01:04.116258 29797 rpcz_store.cc:276] 0501 20:01:02.924513 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.894277 (+969764us) service_pool.cc:225] Handling call
0501 20:01:03.895991 (+ 1714us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:03.896105 (+ 114us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:04.028262 (+132157us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:04.077173 (+ 48911us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:04.078549 (+ 1376us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:04.090284 29725 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:36258 (request call id 53) took 1171 ms. Trace:
I20240501 20:01:04.124572 29643 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59124 (request call id 53) took 1216 ms. Trace:
I20240501 20:01:04.090862 29648 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59124 (request call id 42) took 1222 ms. Trace:
I20240501 20:01:04.090719 29723 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:36258 (request call id 49) took 1227 ms. Trace:
I20240501 20:01:04.095139 29716 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:36258 (request call id 57) took 1076 ms. Trace:
I20240501 20:01:04.099938 29650 rpcz_store.cc:276] 0501 20:01:02.868000 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.855446 (+987446us) service_pool.cc:225] Handling call
0501 20:01:03.857394 (+ 1948us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:03.857485 (+ 91us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:03.982845 (+125360us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:04.040435 (+ 57590us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:04.040851 (+ 416us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:04.067786 29642 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59124 (request call id 44) took 1198 ms. Trace:
I20240501 20:01:04.125142 29851 tablet_bootstrap.cc:654] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:04.111337 29644 rpcz_store.cc:276] 0501 20:01:02.870397 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.873274 (+1002877us) service_pool.cc:225] Handling call
0501 20:01:03.956274 (+ 83000us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:03.956377 (+ 103us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:03.989693 (+ 33316us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:04.058225 (+ 68532us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:04.058700 (+ 475us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":705792}
I20240501 20:01:04.125453 29719 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:36258 (request call id 55) took 1185 ms. Trace:
I20240501 20:01:04.134214 29852 tablet_bootstrap.cc:654] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:04.151721 29723 rpcz_store.cc:276] 0501 20:01:02.863583 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.878005 (+1014422us) service_pool.cc:225] Handling call
0501 20:01:03.980419 (+102414us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:03.980525 (+ 106us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:04.041305 (+ 60780us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:04.087193 (+ 45888us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:04.090478 (+ 3285us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:04.151670 29719 rpcz_store.cc:276] 0501 20:01:02.923813 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:04.057037 (+1133224us) service_pool.cc:225] Handling call
0501 20:01:04.058705 (+ 1668us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:04.058786 (+ 81us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:04.094881 (+ 36095us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:04.109202 (+ 14321us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:04.109650 (+ 448us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:04.151667 29716 rpcz_store.cc:276] 0501 20:01:03.018837 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:04.060354 (+1041517us) service_pool.cc:225] Handling call
0501 20:01:04.062090 (+ 1736us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:04.062180 (+ 90us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:04.089929 (+ 27749us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:04.094778 (+ 4849us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:04.095074 (+ 296us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:04.131546 29643 rpcz_store.cc:276] 0501 20:01:02.908537 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:04.096669 (+1188132us) service_pool.cc:225] Handling call
0501 20:01:04.098700 (+ 2031us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:04.098798 (+ 98us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:04.110924 (+ 12126us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:04.123943 (+ 13019us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:04.124485 (+ 542us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:04.116808 29641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec073fe645614fe28e7837de3c22510b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:04.162683 29793 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41856 (request call id 57) took 1131 ms. Trace:
I20240501 20:01:04.163193 29645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31fff64b453a423eb0f567b3171564fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:04.131596 29648 rpcz_store.cc:276] 0501 20:01:02.868603 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.943690 (+1075087us) service_pool.cc:225] Handling call
0501 20:01:03.988872 (+ 45182us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:03.988976 (+ 104us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:04.050595 (+ 61619us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:04.087323 (+ 36728us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:04.090433 (+ 3110us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:04.163064 29793 rpcz_store.cc:276] 0501 20:01:03.031430 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.961257 (+929827us) service_pool.cc:225] Handling call
0501 20:01:04.048639 (+ 87382us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:04.048737 (+ 98us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:04.155115 (+106378us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:04.162199 (+ 7084us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:04.162628 (+ 429us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:04.165758 29649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77a35893b91247a99912eba787d787cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:04.166792 29647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8787bc4d6b44649b9138d906730110a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:04.172716 29724 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:36258 (request call id 56) took 1174 ms. Trace:
I20240501 20:01:04.173313 29724 rpcz_store.cc:276] 0501 20:01:02.998376 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:04.058267 (+1059891us) service_pool.cc:225] Handling call
0501 20:01:04.059901 (+ 1634us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:04.059983 (+ 82us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:04.146696 (+ 86713us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:04.172062 (+ 25366us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:04.172633 (+ 571us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:04.158581 29644 tablet_service.cc:1467] Processing CreateTablet for tablet 84fa28fb55f44b61ba1a88ed775a78ba (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20240501 20:01:04.175829 29856 raft_consensus.cc:482] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:04.176388 29856 raft_consensus.cc:504] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.178293 29856 leader_election.cc:288] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:04.161880 29642 rpcz_store.cc:276] 0501 20:01:02.869147 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:03.856733 (+987586us) service_pool.cc:225] Handling call
0501 20:01:03.945565 (+ 88832us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:03.945657 (+ 92us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:04.023501 (+ 77844us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:04.067298 (+ 43797us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:04.067723 (+ 425us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:04.160276 29650 tablet_service.cc:1467] Processing CreateTablet for tablet 4e49dd5180f04313ab33942fb3ddc147 (DEFAULT_TABLE table=test-workload [id=c4cd6efa2a67402dac287d449c66135b]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20240501 20:01:04.130096 29849 tablet_bootstrap.cc:492] T 1ca3948261374cef8b1b206a43c00173 P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:04.180421 29849 ts_tablet_manager.cc:1367] T 1ca3948261374cef8b1b206a43c00173 P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.118s user 0.007s sys 0.010s
I20240501 20:01:04.180761 29646 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59124 (request call id 47) took 1310 ms. Trace:
I20240501 20:01:04.181150 29646 rpcz_store.cc:276] 0501 20:01:02.870106 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:04.017373 (+1147267us) service_pool.cc:225] Handling call
0501 20:01:04.112460 (+ 95087us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:04.112560 (+ 100us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:04.157261 (+ 44701us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:04.180294 (+ 23033us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:04.180702 (+ 408us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:04.182250 29641 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59124 (request call id 46) took 1312 ms. Trace:
I20240501 20:01:04.182634 29641 rpcz_store.cc:276] 0501 20:01:02.869705 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:04.003594 (+1133889us) service_pool.cc:225] Handling call
0501 20:01:04.116361 (+112767us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:04.116451 (+ 90us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:04.171516 (+ 55065us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:04.181758 (+ 10242us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:04.182195 (+ 437us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:04.183219 29849 raft_consensus.cc:348] T 1ca3948261374cef8b1b206a43c00173 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.125716 29725 rpcz_store.cc:276] 0501 20:01:02.918794 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:04.011694 (+1092900us) service_pool.cc:225] Handling call
0501 20:01:04.014010 (+ 2316us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:04.014148 (+ 138us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:04.047138 (+ 32990us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:04.089837 (+ 42699us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:04.090224 (+ 387us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:04.184724 29852 tablet_bootstrap.cc:492] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:04.185204 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d2cac25f354a97b1bfb53d57ae7c08" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:04.185181 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d2cac25f354a97b1bfb53d57ae7c08" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:04.185671 29851 tablet_bootstrap.cc:492] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:04.185178 29852 ts_tablet_manager.cc:1367] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.211s user 0.020s sys 0.012s
I20240501 20:01:04.194730 29852 raft_consensus.cc:348] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.195505 29852 raft_consensus.cc:374] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:04.195783 29852 raft_consensus.cc:729] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:04.195994 29645 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59124 (request call id 55) took 1221 ms. Trace:
I20240501 20:01:04.183970 29849 raft_consensus.cc:374] T 1ca3948261374cef8b1b206a43c00173 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:04.196449 29849 raft_consensus.cc:729] T 1ca3948261374cef8b1b206a43c00173 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:04.196540 29647 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59124 (request call id 54) took 1221 ms. Trace:
I20240501 20:01:04.197192 29849 consensus_queue.cc:260] T 1ca3948261374cef8b1b206a43c00173 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.175933 29644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84fa28fb55f44b61ba1a88ed775a78ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:04.186146 29851 ts_tablet_manager.cc:1367] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.074s user 0.011s sys 0.013s
W20240501 20:01:04.186798 29700 leader_election.cc:341] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:04.196918 29647 rpcz_store.cc:276] 0501 20:01:02.974687 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:04.117599 (+1142912us) service_pool.cc:225] Handling call
0501 20:01:04.119427 (+ 1828us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:04.119516 (+ 89us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:04.189936 (+ 70420us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:04.196111 (+ 6175us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:04.196487 (+ 376us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240501 20:01:04.198511 29698 leader_election.cc:341] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:04.200627 29849 ts_tablet_manager.cc:1398] T 1ca3948261374cef8b1b206a43c00173 P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20240501 20:01:04.200799 29851 raft_consensus.cc:348] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.201171 29698 leader_election.cc:302] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:04.201637 29851 raft_consensus.cc:374] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:04.201929 29851 raft_consensus.cc:729] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:04.202800 29650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e49dd5180f04313ab33942fb3ddc147. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:04.204128 29849 tablet_bootstrap.cc:492] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:04.203809 29851 consensus_queue.cc:260] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.196556 29852 consensus_queue.cc:260] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.206491 29852 ts_tablet_manager.cc:1398] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.014s user 0.001s sys 0.004s
I20240501 20:01:04.210075 29849 tablet_bootstrap.cc:654] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:04.210278 29852 tablet_bootstrap.cc:492] T 9046e38fc57f4364aa3d539e1fc01928 P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:04.212667 29644 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59124 (request call id 57) took 1100 ms. Trace:
I20240501 20:01:04.213099 29644 rpcz_store.cc:276] 0501 20:01:03.113391 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:04.153777 (+1040386us) service_pool.cc:225] Handling call
0501 20:01:04.175335 (+ 21558us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:04.175465 (+ 130us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:04.205938 (+ 30473us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:04.212208 (+ 6270us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:04.212611 (+ 403us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:04.197892 29649 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59124 (request call id 56) took 1197 ms. Trace:
I20240501 20:01:04.210381 29645 rpcz_store.cc:276] 0501 20:01:02.974991 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:04.118518 (+1143527us) service_pool.cc:225] Handling call
0501 20:01:04.120540 (+ 2022us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:04.120970 (+ 430us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:04.189187 (+ 68217us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:04.195524 (+ 6337us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:04.195936 (+ 412us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":21376}
I20240501 20:01:04.214345 29649 rpcz_store.cc:276] 0501 20:01:03.000535 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:04.118627 (+1118092us) service_pool.cc:225] Handling call
0501 20:01:04.120730 (+ 2103us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:04.120815 (+ 85us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:04.178640 (+ 57825us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:04.197473 (+ 18833us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:04.197839 (+ 366us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10112}
I20240501 20:01:04.210474 29856 raft_consensus.cc:2726] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:04.229585 29851 ts_tablet_manager.cc:1398] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.031s user 0.013s sys 0.006s
I20240501 20:01:04.230775 29863 raft_consensus.cc:482] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:04.231031 29851 tablet_bootstrap.cc:492] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:04.231281 29863 raft_consensus.cc:504] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.232605 29852 tablet_bootstrap.cc:654] T 9046e38fc57f4364aa3d539e1fc01928 P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:04.233287 29863 leader_election.cc:288] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:04.234997 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d2cac25f354a97b1bfb53d57ae7c08" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:04.237035 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d2cac25f354a97b1bfb53d57ae7c08" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:04.237690 29735 raft_consensus.cc:2454] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 0.
I20240501 20:01:04.238147 29851 tablet_bootstrap.cc:654] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:04.239202 29853 raft_consensus.cc:482] T 1ca3948261374cef8b1b206a43c00173 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:04.239660 29853 raft_consensus.cc:504] T 1ca3948261374cef8b1b206a43c00173 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.240607 29650 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59124 (request call id 58) took 1122 ms. Trace:
I20240501 20:01:04.240967 29650 rpcz_store.cc:276] 0501 20:01:03.118685 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:04.151979 (+1033294us) service_pool.cc:225] Handling call
0501 20:01:04.179667 (+ 27688us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0501 20:01:04.179752 (+ 85us) ts_tablet_manager.cc:655] Creating new metadata...
0501 20:01:04.233127 (+ 53375us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:04.240146 (+ 7019us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:04.240550 (+ 404us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:04.241676 29853 leader_election.cc:288] T 1ca3948261374cef8b1b206a43c00173 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
W20240501 20:01:04.242873 29625 leader_election.cc:341] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:04.243502 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca3948261374cef8b1b206a43c00173" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:04.243721 29624 leader_election.cc:302] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
W20240501 20:01:04.245045 29773 leader_election.cc:341] T 1ca3948261374cef8b1b206a43c00173 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:04.246554 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca3948261374cef8b1b206a43c00173" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
W20240501 20:01:04.247895 29772 leader_election.cc:341] T 1ca3948261374cef8b1b206a43c00173 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:04.248355 29772 leader_election.cc:302] T 1ca3948261374cef8b1b206a43c00173 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:04.249259 29853 raft_consensus.cc:2726] T 1ca3948261374cef8b1b206a43c00173 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:04.249879 29863 raft_consensus.cc:2781] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:04.250169 29863 raft_consensus.cc:482] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:04.250422 29863 raft_consensus.cc:3037] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:04.255021 29863 raft_consensus.cc:504] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.257499 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d2cac25f354a97b1bfb53d57ae7c08" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:04.258064 29735 raft_consensus.cc:3037] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:04.261129 29849 tablet_bootstrap.cc:492] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:04.261633 29849 ts_tablet_manager.cc:1367] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.058s user 0.009s sys 0.017s
I20240501 20:01:04.264465 29849 raft_consensus.cc:348] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.265302 29849 raft_consensus.cc:374] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:04.265622 29849 raft_consensus.cc:729] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:04.265856 29735 raft_consensus.cc:2454] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:04.266532 29849 consensus_queue.cc:260] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.266975 29624 leader_election.cc:302] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:04.268991 29849 ts_tablet_manager.cc:1398] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20240501 20:01:04.270793 29849 tablet_bootstrap.cc:492] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:04.272605 29851 tablet_bootstrap.cc:492] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:04.273263 29863 leader_election.cc:288] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:04.275885 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d2cac25f354a97b1bfb53d57ae7c08" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:04.276387 29863 raft_consensus.cc:2781] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:04.277266 29849 tablet_bootstrap.cc:654] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:04.277746 29863 raft_consensus.cc:686] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
W20240501 20:01:04.278312 29625 leader_election.cc:341] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:04.279541 29863 consensus_queue.cc:237] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.275178 29852 tablet_bootstrap.cc:492] T 9046e38fc57f4364aa3d539e1fc01928 P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:04.273716 29851 ts_tablet_manager.cc:1367] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.043s user 0.020s sys 0.021s
I20240501 20:01:04.286716 29852 ts_tablet_manager.cc:1367] T 9046e38fc57f4364aa3d539e1fc01928 P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.077s user 0.018s sys 0.029s
I20240501 20:01:04.289059 29851 raft_consensus.cc:348] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.289494 29852 raft_consensus.cc:348] T 9046e38fc57f4364aa3d539e1fc01928 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.290112 29851 raft_consensus.cc:374] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:04.290225 29852 raft_consensus.cc:374] T 9046e38fc57f4364aa3d539e1fc01928 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:04.291297 29851 raft_consensus.cc:729] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:04.291373 29852 raft_consensus.cc:729] T 9046e38fc57f4364aa3d539e1fc01928 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:04.292501 29851 consensus_queue.cc:260] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.292586 29852 consensus_queue.cc:260] T 9046e38fc57f4364aa3d539e1fc01928 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.296257 29851 ts_tablet_manager.cc:1398] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.011s user 0.009s sys 0.001s
I20240501 20:01:04.297575 29851 tablet_bootstrap.cc:492] T 9046e38fc57f4364aa3d539e1fc01928 P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:04.300891 29883 raft_consensus.cc:482] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:04.301396 29883 raft_consensus.cc:504] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } }
I20240501 20:01:04.304543 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69915dd5992641e3a24f7f45f1cd5c0c" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:04.305953 29625 leader_election.cc:341] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:04.306772 29851 tablet_bootstrap.cc:654] T 9046e38fc57f4364aa3d539e1fc01928 P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:04.310626 29849 tablet_bootstrap.cc:492] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:04.311249 29849 ts_tablet_manager.cc:1367] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.041s user 0.012s sys 0.018s
I20240501 20:01:04.314836 29849 raft_consensus.cc:348] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.317201 29883 leader_election.cc:288] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:04.315832 29849 raft_consensus.cc:374] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:04.318059 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69915dd5992641e3a24f7f45f1cd5c0c" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:04.318863 29735 raft_consensus.cc:2454] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 0.
I20240501 20:01:04.319993 29852 ts_tablet_manager.cc:1398] T 9046e38fc57f4364aa3d539e1fc01928 P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.033s user 0.015s sys 0.004s
I20240501 20:01:04.320080 29624 leader_election.cc:302] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:04.321105 29852 tablet_bootstrap.cc:492] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:04.321565 29851 tablet_bootstrap.cc:492] T 9046e38fc57f4364aa3d539e1fc01928 P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:04.318053 29849 raft_consensus.cc:729] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:04.322543 29887 raft_consensus.cc:2781] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:04.322954 29887 raft_consensus.cc:482] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:04.323251 29849 consensus_queue.cc:260] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.324422 29887 raft_consensus.cc:3037] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:04.327453 29852 tablet_bootstrap.cc:654] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:04.328356 29849 ts_tablet_manager.cc:1398] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.017s user 0.009s sys 0.000s
I20240501 20:01:04.328518 29853 raft_consensus.cc:482] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:04.328959 29853 raft_consensus.cc:504] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.329471 29849 tablet_bootstrap.cc:492] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:04.331061 29853 leader_election.cc:288] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:04.322053 29851 ts_tablet_manager.cc:1367] T 9046e38fc57f4364aa3d539e1fc01928 P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.025s user 0.013s sys 0.007s
I20240501 20:01:04.320217 29556 catalog_manager.cc:5484] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 1 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:04.332995 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9046e38fc57f4364aa3d539e1fc01928" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:04.333907 29660 raft_consensus.cc:2454] T 9046e38fc57f4364aa3d539e1fc01928 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 0.
I20240501 20:01:04.335314 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9046e38fc57f4364aa3d539e1fc01928" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:04.336467 29849 tablet_bootstrap.cc:654] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:04.336890 29772 leader_election.cc:302] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:04.338421 29853 raft_consensus.cc:2781] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:04.340066 29853 raft_consensus.cc:482] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:04.340616 29853 raft_consensus.cc:3037] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:04.338575 29851 raft_consensus.cc:348] T 9046e38fc57f4364aa3d539e1fc01928 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.356014 29851 raft_consensus.cc:374] T 9046e38fc57f4364aa3d539e1fc01928 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:04.356537 29851 raft_consensus.cc:729] T 9046e38fc57f4364aa3d539e1fc01928 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:04.354816 29853 raft_consensus.cc:504] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.361218 29852 tablet_bootstrap.cc:492] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:04.361807 29852 ts_tablet_manager.cc:1367] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.041s user 0.022s sys 0.015s
I20240501 20:01:04.363755 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9046e38fc57f4364aa3d539e1fc01928" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:04.364368 29660 raft_consensus.cc:3037] T 9046e38fc57f4364aa3d539e1fc01928 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:04.364909 29852 raft_consensus.cc:348] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.365653 29852 raft_consensus.cc:374] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:04.365940 29852 raft_consensus.cc:729] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:04.366607 29852 consensus_queue.cc:260] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.368136 29887 raft_consensus.cc:504] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } }
I20240501 20:01:04.368810 29852 ts_tablet_manager.cc:1398] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240501 20:01:04.369822 29852 tablet_bootstrap.cc:492] T 1ca3948261374cef8b1b206a43c00173 P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:04.371862 29660 raft_consensus.cc:2454] T 9046e38fc57f4364aa3d539e1fc01928 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
I20240501 20:01:04.371762 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69915dd5992641e3a24f7f45f1cd5c0c" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:04.373673 29772 leader_election.cc:302] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:04.374754 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9046e38fc57f4364aa3d539e1fc01928" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:04.374583 29851 consensus_queue.cc:260] T 9046e38fc57f4364aa3d539e1fc01928 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:04.375141 29625 leader_election.cc:341] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:04.375627 29735 raft_consensus.cc:2454] T 9046e38fc57f4364aa3d539e1fc01928 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 0.
I20240501 20:01:04.377465 29883 raft_consensus.cc:482] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:04.378057 29883 raft_consensus.cc:504] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.380373 29852 tablet_bootstrap.cc:654] T 1ca3948261374cef8b1b206a43c00173 P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:04.380627 29883 leader_election.cc:288] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:04.382289 29851 ts_tablet_manager.cc:1398] T 9046e38fc57f4364aa3d539e1fc01928 P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.047s user 0.002s sys 0.004s
I20240501 20:01:04.382850 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3f29f0eeea4b119984634202e5d9cd" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:04.383381 29851 tablet_bootstrap.cc:492] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
W20240501 20:01:04.384578 29624 leader_election.cc:341] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:04.388154 29853 leader_election.cc:288] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:04.389003 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69915dd5992641e3a24f7f45f1cd5c0c" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:04.390044 29853 raft_consensus.cc:2781] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:04.390326 29735 raft_consensus.cc:3037] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:04.390556 29853 raft_consensus.cc:686] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:04.391820 29852 tablet_bootstrap.cc:492] T 1ca3948261374cef8b1b206a43c00173 P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:04.391465 29853 consensus_queue.cc:237] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.392292 29852 ts_tablet_manager.cc:1367] T 1ca3948261374cef8b1b206a43c00173 P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.007s
I20240501 20:01:04.394325 29887 leader_election.cc:288] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Requested vote from peers ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:04.395074 29852 raft_consensus.cc:348] T 1ca3948261374cef8b1b206a43c00173 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.395859 29849 tablet_bootstrap.cc:492] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:04.395790 29852 raft_consensus.cc:374] T 1ca3948261374cef8b1b206a43c00173 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:04.396260 29852 raft_consensus.cc:729] T 1ca3948261374cef8b1b206a43c00173 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:04.396370 29849 ts_tablet_manager.cc:1367] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.067s user 0.014s sys 0.041s
I20240501 20:01:04.396951 29852 consensus_queue.cc:260] T 1ca3948261374cef8b1b206a43c00173 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.399010 29735 raft_consensus.cc:2454] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:04.398990 29849 raft_consensus.cc:348] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.399946 29849 raft_consensus.cc:374] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:04.400326 29849 raft_consensus.cc:729] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:04.400885 29624 leader_election.cc:302] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:04.401146 29849 consensus_queue.cc:260] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.399091 29852 ts_tablet_manager.cc:1398] T 1ca3948261374cef8b1b206a43c00173 P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240501 20:01:04.404325 29852 tablet_bootstrap.cc:492] T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:04.401978 29887 raft_consensus.cc:2781] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:04.405277 29887 raft_consensus.cc:686] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:04.406208 29887 consensus_queue.cc:237] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } }
I20240501 20:01:04.407418 29851 tablet_bootstrap.cc:654] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:04.410508 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3f29f0eeea4b119984634202e5d9cd" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:04.410908 29852 tablet_bootstrap.cc:654] T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:04.411231 29849 ts_tablet_manager.cc:1398] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.015s user 0.008s sys 0.002s
W20240501 20:01:04.412122 29625 leader_election.cc:341] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:04.412343 29849 tablet_bootstrap.cc:492] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:04.412596 29625 leader_election.cc:302] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:04.413786 29883 raft_consensus.cc:2726] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:04.436178 29849 tablet_bootstrap.cc:654] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:04.440665 29851 tablet_bootstrap.cc:492] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:04.441190 29851 ts_tablet_manager.cc:1367] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.058s user 0.013s sys 0.011s
I20240501 20:01:04.443883 29851 raft_consensus.cc:348] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.444641 29851 raft_consensus.cc:374] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:04.444898 29851 raft_consensus.cc:729] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:04.445673 29851 consensus_queue.cc:260] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.448088 29851 ts_tablet_manager.cc:1398] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.007s user 0.001s sys 0.006s
I20240501 20:01:04.450083 29851 tablet_bootstrap.cc:492] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:04.449654 29558 catalog_manager.cc:5484] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 0 to 1, leader changed from <none> to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 1 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:04.456583 29851 tablet_bootstrap.cc:654] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:04.463644 29852 tablet_bootstrap.cc:492] T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:04.463882 29849 tablet_bootstrap.cc:492] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:04.463161 29555 catalog_manager.cc:5484] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 1 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:04.464534 29849 ts_tablet_manager.cc:1367] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.052s user 0.013s sys 0.004s
I20240501 20:01:04.464164 29852 ts_tablet_manager.cc:1367] T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.060s user 0.028s sys 0.020s
I20240501 20:01:04.466804 29849 raft_consensus.cc:348] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.467471 29849 raft_consensus.cc:374] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:04.467727 29849 raft_consensus.cc:729] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:04.468152 29853 raft_consensus.cc:482] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:04.468407 29849 consensus_queue.cc:260] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.468618 29853 raft_consensus.cc:504] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.470386 29849 ts_tablet_manager.cc:1398] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240501 20:01:04.470655 29853 leader_election.cc:288] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:04.471207 29849 tablet_bootstrap.cc:492] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:04.472748 29851 tablet_bootstrap.cc:492] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:04.473086 29851 ts_tablet_manager.cc:1367] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.023s user 0.014s sys 0.000s
I20240501 20:01:04.473737 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49da4246f5f6438abe0e7e8c9a523d80" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:04.473752 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49da4246f5f6438abe0e7e8c9a523d80" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:04.474632 29735 raft_consensus.cc:2454] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 0.
I20240501 20:01:04.475458 29851 raft_consensus.cc:348] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.476188 29851 raft_consensus.cc:374] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:04.475979 29773 leader_election.cc:302] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
W20240501 20:01:04.476630 29772 leader_election.cc:341] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:04.476779 29851 raft_consensus.cc:729] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:04.476914 29896 raft_consensus.cc:2781] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:04.477445 29896 raft_consensus.cc:482] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:04.467690 29852 raft_consensus.cc:348] T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.477851 29852 raft_consensus.cc:374] T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:04.477795 29896 raft_consensus.cc:3037] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:04.478408 29852 raft_consensus.cc:729] T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:04.477806 29851 consensus_queue.cc:260] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.479235 29852 consensus_queue.cc:260] T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.480417 29851 ts_tablet_manager.cc:1398] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20240501 20:01:04.481294 29851 tablet_bootstrap.cc:492] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:04.481498 29852 ts_tablet_manager.cc:1398] T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20240501 20:01:04.482434 29852 tablet_bootstrap.cc:492] T add9d7579351483fb268f89a44e95d8a P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:04.476783 29849 tablet_bootstrap.cc:654] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:04.486296 29851 tablet_bootstrap.cc:654] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:04.486052 29896 raft_consensus.cc:504] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.489069 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49da4246f5f6438abe0e7e8c9a523d80" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:04.489394 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49da4246f5f6438abe0e7e8c9a523d80" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:04.490005 29735 raft_consensus.cc:3037] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:04.489074 29896 leader_election.cc:288] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:04.490800 29852 tablet_bootstrap.cc:654] T add9d7579351483fb268f89a44e95d8a P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
W20240501 20:01:04.492733 29772 leader_election.cc:341] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:04.494576 29851 tablet_bootstrap.cc:492] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:04.495028 29851 ts_tablet_manager.cc:1367] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.008s
I20240501 20:01:04.495507 29849 tablet_bootstrap.cc:492] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:04.495970 29849 ts_tablet_manager.cc:1367] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.025s user 0.010s sys 0.000s
I20240501 20:01:04.496400 29887 raft_consensus.cc:482] T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:04.496842 29887 raft_consensus.cc:504] T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.497558 29851 raft_consensus.cc:348] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.498256 29851 raft_consensus.cc:374] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:04.498279 29849 raft_consensus.cc:348] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.498796 29887 leader_election.cc:288] T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:04.499125 29851 raft_consensus.cc:729] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:04.499230 29849 raft_consensus.cc:374] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:04.499666 29849 raft_consensus.cc:729] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:04.499540 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d88139f89c47ff84cad674147aae0f" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:04.499987 29851 consensus_queue.cc:260] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.500602 29849 consensus_queue.cc:260] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:04.501390 29624 leader_election.cc:341] T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:04.501606 29852 tablet_bootstrap.cc:492] T add9d7579351483fb268f89a44e95d8a P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:04.502075 29852 ts_tablet_manager.cc:1367] T add9d7579351483fb268f89a44e95d8a P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.004s
I20240501 20:01:04.502338 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d88139f89c47ff84cad674147aae0f" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:04.503015 29849 ts_tablet_manager.cc:1398] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240501 20:01:04.502987 29809 raft_consensus.cc:2454] T f6d88139f89c47ff84cad674147aae0f P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 0.
I20240501 20:01:04.504073 29849 tablet_bootstrap.cc:492] T d1d2cac25f354a97b1bfb53d57ae7c08 P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:04.504091 29625 leader_election.cc:302] T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters: fdfeac4190e045f290e003b2298d988d
I20240501 20:01:04.504753 29852 raft_consensus.cc:348] T add9d7579351483fb268f89a44e95d8a P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.505261 29887 raft_consensus.cc:2781] T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:04.505388 29852 raft_consensus.cc:374] T add9d7579351483fb268f89a44e95d8a P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:04.505676 29887 raft_consensus.cc:482] T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:04.505721 29852 raft_consensus.cc:729] T add9d7579351483fb268f89a44e95d8a P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:04.506134 29887 raft_consensus.cc:3037] T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:04.506462 29852 consensus_queue.cc:260] T add9d7579351483fb268f89a44e95d8a P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.508796 29852 ts_tablet_manager.cc:1398] T add9d7579351483fb268f89a44e95d8a P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20240501 20:01:04.509512 29849 tablet_bootstrap.cc:654] T d1d2cac25f354a97b1bfb53d57ae7c08 P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:04.509619 29852 tablet_bootstrap.cc:492] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:04.515202 29852 tablet_bootstrap.cc:654] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:04.502329 29851 ts_tablet_manager.cc:1398] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240501 20:01:04.516920 29851 tablet_bootstrap.cc:492] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:04.513832 29887 raft_consensus.cc:504] T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.519733 29879 raft_consensus.cc:482] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:04.520372 29879 raft_consensus.cc:504] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.521931 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d88139f89c47ff84cad674147aae0f" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:04.522732 29809 raft_consensus.cc:3037] T f6d88139f89c47ff84cad674147aae0f P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:04.526770 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668b4a4d782c4278932dd8413dc3ab3e" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:04.527799 29879 leader_election.cc:288] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:04.529496 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668b4a4d782c4278932dd8413dc3ab3e" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:04.538934 29808 raft_consensus.cc:2454] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 0.
I20240501 20:01:04.533035 29896 raft_consensus.cc:482] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:04.533492 29887 leader_election.cc:288] T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:04.534816 29698 leader_election.cc:341] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:04.537801 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d88139f89c47ff84cad674147aae0f" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:04.540117 29896 raft_consensus.cc:504] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:04.541736 29624 leader_election.cc:341] T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:04.538004 29809 raft_consensus.cc:2454] T f6d88139f89c47ff84cad674147aae0f P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:04.544090 29896 leader_election.cc:288] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:04.544651 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add9d7579351483fb268f89a44e95d8a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:04.545419 29660 raft_consensus.cc:2454] T add9d7579351483fb268f89a44e95d8a P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 0.
I20240501 20:01:04.545792 29625 leader_election.cc:302] T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters: fdfeac4190e045f290e003b2298d988d
I20240501 20:01:04.546746 29772 leader_election.cc:302] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:04.547940 29896 raft_consensus.cc:2781] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:04.548786 29896 raft_consensus.cc:482] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:04.548625 29700 leader_election.cc:302] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:04.549427 29896 raft_consensus.cc:3037] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:04.550002 29856 raft_consensus.cc:2781] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:04.550514 29856 raft_consensus.cc:482] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:04.550932 29856 raft_consensus.cc:3037] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:04.557623 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add9d7579351483fb268f89a44e95d8a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:04.558588 29734 raft_consensus.cc:2454] T add9d7579351483fb268f89a44e95d8a P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 0.
I20240501 20:01:04.560528 29851 tablet_bootstrap.cc:654] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:04.561311 29887 raft_consensus.cc:2781] T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:04.563814 29887 raft_consensus.cc:686] T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:04.559279 29896 raft_consensus.cc:504] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.562664 29879 raft_consensus.cc:482] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:04.565287 29879 raft_consensus.cc:504] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.567875 29879 leader_election.cc:288] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:04.569417 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3f29f0eeea4b119984634202e5d9cd" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:04.571023 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add9d7579351483fb268f89a44e95d8a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
W20240501 20:01:04.571301 29700 leader_election.cc:341] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:04.571828 29660 raft_consensus.cc:3037] T add9d7579351483fb268f89a44e95d8a P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:04.573243 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3f29f0eeea4b119984634202e5d9cd" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:04.573956 29659 raft_consensus.cc:2454] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 0.
I20240501 20:01:04.565824 29887 consensus_queue.cc:237] T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.575178 29698 leader_election.cc:302] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:04.581588 29660 raft_consensus.cc:2454] T add9d7579351483fb268f89a44e95d8a P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
I20240501 20:01:04.584779 29879 raft_consensus.cc:2781] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:04.585265 29879 raft_consensus.cc:482] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:04.585680 29879 raft_consensus.cc:3037] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:04.586298 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add9d7579351483fb268f89a44e95d8a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:04.587272 29734 raft_consensus.cc:3037] T add9d7579351483fb268f89a44e95d8a P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:04.588896 29772 leader_election.cc:302] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:04.590123 29853 raft_consensus.cc:2781] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:04.590761 29853 raft_consensus.cc:686] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:04.591862 29853 consensus_queue.cc:237] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.598053 29735 raft_consensus.cc:2454] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
I20240501 20:01:04.598557 29892 raft_consensus.cc:482] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:04.607076 29856 raft_consensus.cc:504] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.613049 29892 raft_consensus.cc:504] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.617874 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668b4a4d782c4278932dd8413dc3ab3e" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:04.619097 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d86f8848a184a87947fd927c44e5a35" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:04.620457 29892 leader_election.cc:288] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
W20240501 20:01:04.621781 29772 leader_election.cc:341] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
W20240501 20:01:04.622815 29698 leader_election.cc:341] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:04.623278 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668b4a4d782c4278932dd8413dc3ab3e" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:04.624140 29809 raft_consensus.cc:3037] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:04.624670 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d86f8848a184a87947fd927c44e5a35" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
W20240501 20:01:04.626856 29773 leader_election.cc:341] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:04.627475 29773 leader_election.cc:302] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:04.629731 29734 raft_consensus.cc:2454] T add9d7579351483fb268f89a44e95d8a P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
I20240501 20:01:04.627056 29856 leader_election.cc:288] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:04.640977 29892 raft_consensus.cc:2726] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:04.645524 29773 leader_election.cc:302] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:04.647326 29892 raft_consensus.cc:2781] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:04.643975 29809 raft_consensus.cc:2454] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
I20240501 20:01:04.610330 29896 leader_election.cc:288] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:04.657061 29700 leader_election.cc:302] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:04.658408 29856 raft_consensus.cc:2781] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:04.660331 29909 raft_consensus.cc:482] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:04.660768 29909 raft_consensus.cc:504] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.664146 29892 raft_consensus.cc:686] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:04.666522 29892 consensus_queue.cc:237] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.667254 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfac893b4b4f457694a4eb184b6abb53" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:04.667850 29659 raft_consensus.cc:2454] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 0.
I20240501 20:01:04.669214 29698 leader_election.cc:302] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:04.663446 29909 leader_election.cc:288] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:04.671407 29856 raft_consensus.cc:686] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:04.662236 29557 catalog_manager.cc:5484] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 0 to 1, leader changed from <none> to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 1 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:04.674516 29909 raft_consensus.cc:2781] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:04.674866 29909 raft_consensus.cc:482] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:04.675174 29909 raft_consensus.cc:3037] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:04.675225 29856 consensus_queue.cc:237] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.668959 29555 catalog_manager.cc:5484] T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 1 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:04.688501 29909 raft_consensus.cc:504] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.690544 29909 leader_election.cc:288] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:04.692081 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfac893b4b4f457694a4eb184b6abb53" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:04.692919 29808 raft_consensus.cc:3037] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:04.695137 29852 tablet_bootstrap.cc:492] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:04.695513 29852 ts_tablet_manager.cc:1367] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.186s user 0.071s sys 0.069s
I20240501 20:01:04.697774 29852 raft_consensus.cc:348] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.698479 29852 raft_consensus.cc:374] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:04.698733 29852 raft_consensus.cc:729] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:04.699407 29852 consensus_queue.cc:260] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.704290 29851 tablet_bootstrap.cc:492] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:04.704946 29851 ts_tablet_manager.cc:1367] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.188s user 0.009s sys 0.011s
I20240501 20:01:04.707804 29852 ts_tablet_manager.cc:1398] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20240501 20:01:04.701186 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfac893b4b4f457694a4eb184b6abb53" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:04.708781 29852 tablet_bootstrap.cc:492] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:04.709069 29849 tablet_bootstrap.cc:492] T d1d2cac25f354a97b1bfb53d57ae7c08 P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:04.709519 29849 ts_tablet_manager.cc:1367] T d1d2cac25f354a97b1bfb53d57ae7c08 P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.206s user 0.063s sys 0.087s
I20240501 20:01:04.711982 29849 raft_consensus.cc:348] T d1d2cac25f354a97b1bfb53d57ae7c08 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.712692 29849 raft_consensus.cc:374] T d1d2cac25f354a97b1bfb53d57ae7c08 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:04.712961 29849 raft_consensus.cc:729] T d1d2cac25f354a97b1bfb53d57ae7c08 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:04.713670 29849 consensus_queue.cc:260] T d1d2cac25f354a97b1bfb53d57ae7c08 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.714629 29851 raft_consensus.cc:348] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:04.735363 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 66217.8us
I20240501 20:01:04.735648 29851 raft_consensus.cc:374] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:04.735965 29851 raft_consensus.cc:729] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:04.736577 29852 tablet_bootstrap.cc:654] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:04.737057 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfac893b4b4f457694a4eb184b6abb53" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:04.736608 29851 consensus_queue.cc:260] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.737632 29659 raft_consensus.cc:3037] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:04.738631 29851 ts_tablet_manager.cc:1398] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.033s user 0.005s sys 0.001s
I20240501 20:01:04.739568 29851 tablet_bootstrap.cc:492] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:04.716941 29557 catalog_manager.cc:5484] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 0 to 1, leader changed from <none> to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 1 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:04.745090 29849 ts_tablet_manager.cc:1398] T d1d2cac25f354a97b1bfb53d57ae7c08 P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.035s user 0.009s sys 0.001s
I20240501 20:01:04.728482 29879 raft_consensus.cc:504] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.747623 29849 tablet_bootstrap.cc:492] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:04.716078 29808 raft_consensus.cc:2454] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
I20240501 20:01:04.749775 29700 leader_election.cc:302] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:04.750231 29555 catalog_manager.cc:5484] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 0 to 1, leader changed from <none> to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 1 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:04.752530 29849 tablet_bootstrap.cc:654] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:04.745986 29659 raft_consensus.cc:2454] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
I20240501 20:01:04.753125 29851 tablet_bootstrap.cc:654] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:04.755187 29856 raft_consensus.cc:2781] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:04.755709 29856 raft_consensus.cc:686] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:04.756215 29852 tablet_bootstrap.cc:492] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:04.757241 29852 ts_tablet_manager.cc:1367] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.049s user 0.013s sys 0.004s
I20240501 20:01:04.758860 29887 raft_consensus.cc:482] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:04.759315 29887 raft_consensus.cc:504] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.761258 29887 leader_election.cc:288] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:04.758046 29856 consensus_queue.cc:237] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.763125 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668b4a4d782c4278932dd8413dc3ab3e" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:04.764819 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668b4a4d782c4278932dd8413dc3ab3e" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:04.765614 29808 raft_consensus.cc:2379] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 1: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:04.766721 29625 leader_election.cc:302] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:04.766739 29852 raft_consensus.cc:348] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.767621 29852 raft_consensus.cc:374] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:04.767920 29887 raft_consensus.cc:3037] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:04.767903 29852 raft_consensus.cc:729] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:04.768765 29879 leader_election.cc:288] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:04.769866 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3f29f0eeea4b119984634202e5d9cd" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:04.771152 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3f29f0eeea4b119984634202e5d9cd" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:04.768895 29852 consensus_queue.cc:260] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.771764 29659 raft_consensus.cc:3037] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
W20240501 20:01:04.772272 29700 leader_election.cc:341] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:04.775764 29887 raft_consensus.cc:2726] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:04.776237 29851 tablet_bootstrap.cc:492] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:04.776809 29851 ts_tablet_manager.cc:1367] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.037s user 0.009s sys 0.007s
I20240501 20:01:04.779268 29851 raft_consensus.cc:348] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.780030 29851 raft_consensus.cc:374] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:04.780311 29851 raft_consensus.cc:729] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:04.780596 29659 raft_consensus.cc:2454] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
I20240501 20:01:04.780987 29851 consensus_queue.cc:260] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.781683 29698 leader_election.cc:302] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:04.782464 29879 raft_consensus.cc:2781] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:04.783020 29879 raft_consensus.cc:686] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:04.784125 29914 raft_consensus.cc:482] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:04.783771 29879 consensus_queue.cc:237] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.784595 29914 raft_consensus.cc:504] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.786334 29914 leader_election.cc:288] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:04.787668 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05df43bdfa479b902b1da297fb5974" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:04.787668 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05df43bdfa479b902b1da297fb5974" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:04.787318 29555 catalog_manager.cc:5484] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 0 to 1, leader changed from <none> to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 1 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:04.788614 29808 raft_consensus.cc:2454] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 0.
W20240501 20:01:04.789469 29698 leader_election.cc:341] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:04.790000 29700 leader_election.cc:302] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:04.790786 29849 tablet_bootstrap.cc:492] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:04.790899 29916 raft_consensus.cc:2781] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:04.791193 29849 ts_tablet_manager.cc:1367] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.044s user 0.009s sys 0.000s
I20240501 20:01:04.791313 29916 raft_consensus.cc:482] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:04.791707 29916 raft_consensus.cc:3037] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:04.788084 29851 ts_tablet_manager.cc:1398] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20240501 20:01:04.792809 29851 tablet_bootstrap.cc:492] T 1ca3948261374cef8b1b206a43c00173 P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:04.793566 29849 raft_consensus.cc:348] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } }
I20240501 20:01:04.794258 29849 raft_consensus.cc:374] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:04.794538 29849 raft_consensus.cc:729] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:04.795213 29849 consensus_queue.cc:260] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } }
I20240501 20:01:04.797291 29735 raft_consensus.cc:3037] T 9046e38fc57f4364aa3d539e1fc01928 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:04.798724 29916 raft_consensus.cc:504] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.801378 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05df43bdfa479b902b1da297fb5974" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:04.801643 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05df43bdfa479b902b1da297fb5974" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:04.802239 29808 raft_consensus.cc:3037] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:04.802459 29851 tablet_bootstrap.cc:654] T 1ca3948261374cef8b1b206a43c00173 P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
W20240501 20:01:04.802788 29698 leader_election.cc:341] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:04.809398 29808 raft_consensus.cc:2454] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
I20240501 20:01:04.810150 29892 consensus_queue.cc:1035] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:01:04.811105 29916 leader_election.cc:288] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:04.810479 29700 leader_election.cc:302] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:04.782523 29852 ts_tablet_manager.cc:1398] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.025s user 0.006s sys 0.000s
I20240501 20:01:04.812494 29916 raft_consensus.cc:2781] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:04.812928 29916 raft_consensus.cc:686] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:04.813361 29852 tablet_bootstrap.cc:492] T cb05df43bdfa479b902b1da297fb5974 P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:04.813699 29916 consensus_queue.cc:237] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.818720 29852 tablet_bootstrap.cc:654] T cb05df43bdfa479b902b1da297fb5974 P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:04.818969 29851 tablet_bootstrap.cc:492] T 1ca3948261374cef8b1b206a43c00173 P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:04.819409 29851 ts_tablet_manager.cc:1367] T 1ca3948261374cef8b1b206a43c00173 P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.027s user 0.016s sys 0.001s
I20240501 20:01:04.822177 29851 raft_consensus.cc:348] T 1ca3948261374cef8b1b206a43c00173 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.822878 29851 raft_consensus.cc:374] T 1ca3948261374cef8b1b206a43c00173 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:04.823171 29851 raft_consensus.cc:729] T 1ca3948261374cef8b1b206a43c00173 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:04.824097 29852 tablet_bootstrap.cc:492] T cb05df43bdfa479b902b1da297fb5974 P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:04.823765 29851 consensus_queue.cc:260] T 1ca3948261374cef8b1b206a43c00173 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.814182 29849 ts_tablet_manager.cc:1398] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.023s user 0.007s sys 0.000s
I20240501 20:01:04.824520 29852 ts_tablet_manager.cc:1367] T cb05df43bdfa479b902b1da297fb5974 P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20240501 20:01:04.826964 29852 raft_consensus.cc:348] T cb05df43bdfa479b902b1da297fb5974 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.827646 29852 raft_consensus.cc:374] T cb05df43bdfa479b902b1da297fb5974 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:04.827930 29852 raft_consensus.cc:729] T cb05df43bdfa479b902b1da297fb5974 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:04.828569 29849 tablet_bootstrap.cc:492] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:04.828760 29852 consensus_queue.cc:260] T cb05df43bdfa479b902b1da297fb5974 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.833391 29851 ts_tablet_manager.cc:1398] T 1ca3948261374cef8b1b206a43c00173 P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.014s user 0.003s sys 0.004s
I20240501 20:01:04.834347 29851 tablet_bootstrap.cc:492] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:04.834466 29896 raft_consensus.cc:482] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:04.826313 29555 catalog_manager.cc:5484] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 0 to 1, leader changed from <none> to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 1 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:04.834995 29913 consensus_queue.cc:1035] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:01:04.834882 29896 raft_consensus.cc:504] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } }
I20240501 20:01:04.841553 29896 leader_election.cc:288] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:04.842993 29887 consensus_queue.cc:1035] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:01:04.843540 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69915dd5992641e3a24f7f45f1cd5c0c" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:04.854631 29852 ts_tablet_manager.cc:1398] T cb05df43bdfa479b902b1da297fb5974 P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.030s user 0.004s sys 0.002s
I20240501 20:01:04.855383 29852 tablet_bootstrap.cc:492] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:04.856813 29808 raft_consensus.cc:3037] T d1d2cac25f354a97b1bfb53d57ae7c08 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:04.858536 29887 raft_consensus.cc:482] T 1ca3948261374cef8b1b206a43c00173 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:04.864818 29887 raft_consensus.cc:504] T 1ca3948261374cef8b1b206a43c00173 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.866748 29887 leader_election.cc:288] T 1ca3948261374cef8b1b206a43c00173 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:04.869274 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca3948261374cef8b1b206a43c00173" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:04.870323 29809 raft_consensus.cc:2454] T 1ca3948261374cef8b1b206a43c00173 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 0.
I20240501 20:01:04.871496 29625 leader_election.cc:302] T 1ca3948261374cef8b1b206a43c00173 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:04.872325 29887 raft_consensus.cc:2781] T 1ca3948261374cef8b1b206a43c00173 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:04.872669 29887 raft_consensus.cc:482] T 1ca3948261374cef8b1b206a43c00173 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:04.872982 29887 raft_consensus.cc:3037] T 1ca3948261374cef8b1b206a43c00173 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:04.874522 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca3948261374cef8b1b206a43c00173" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:04.876343 29732 raft_consensus.cc:2454] T 1ca3948261374cef8b1b206a43c00173 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 0.
I20240501 20:01:04.878139 29883 consensus_queue.cc:1035] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:01:04.862709 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69915dd5992641e3a24f7f45f1cd5c0c" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:04.880656 29734 raft_consensus.cc:2379] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 1: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:04.881767 29773 leader_election.cc:302] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:04.882903 29853 raft_consensus.cc:3037] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:04.892966 29887 raft_consensus.cc:504] T 1ca3948261374cef8b1b206a43c00173 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.919472 29887 leader_election.cc:288] T 1ca3948261374cef8b1b206a43c00173 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:04.920259 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca3948261374cef8b1b206a43c00173" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:04.921182 29809 raft_consensus.cc:3037] T 1ca3948261374cef8b1b206a43c00173 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:04.930475 29809 raft_consensus.cc:2454] T 1ca3948261374cef8b1b206a43c00173 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:04.931608 29625 leader_election.cc:302] T 1ca3948261374cef8b1b206a43c00173 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:04.931396 29887 consensus_queue.cc:1035] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:01:04.932541 29883 raft_consensus.cc:2781] T 1ca3948261374cef8b1b206a43c00173 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:04.937381 29853 raft_consensus.cc:2726] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:04.927855 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca3948261374cef8b1b206a43c00173" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:04.943812 29849 tablet_bootstrap.cc:654] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:04.946108 29851 tablet_bootstrap.cc:654] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:04.947438 29852 tablet_bootstrap.cc:654] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:04.948335 29734 raft_consensus.cc:3037] T 1ca3948261374cef8b1b206a43c00173 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:04.948585 29883 raft_consensus.cc:686] T 1ca3948261374cef8b1b206a43c00173 P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:04.977154 29883 consensus_queue.cc:237] T 1ca3948261374cef8b1b206a43c00173 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:04.987160 29853 consensus_queue.cc:1035] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:01:04.987996 29606 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240501 20:01:04.987655 29887 consensus_queue.cc:1035] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:01:04.987670 29734 raft_consensus.cc:2454] T 1ca3948261374cef8b1b206a43c00173 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:05.014964 29851 tablet_bootstrap.cc:492] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:05.015514 29851 ts_tablet_manager.cc:1367] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.181s user 0.031s sys 0.043s
I20240501 20:01:05.018370 29851 raft_consensus.cc:348] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.019377 29851 raft_consensus.cc:374] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:05.019716 29851 raft_consensus.cc:729] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:05.020601 29851 consensus_queue.cc:260] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.020735 29557 catalog_manager.cc:5484] T 1ca3948261374cef8b1b206a43c00173 P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 1 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:05.022730 29851 ts_tablet_manager.cc:1398] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20240501 20:01:05.023831 29851 tablet_bootstrap.cc:492] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:05.028076 29852 tablet_bootstrap.cc:492] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:05.028554 29852 ts_tablet_manager.cc:1367] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.173s user 0.026s sys 0.040s
I20240501 20:01:05.030898 29852 raft_consensus.cc:348] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.031477 29852 raft_consensus.cc:374] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:05.031764 29852 raft_consensus.cc:729] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:05.032296 29896 consensus_queue.cc:1035] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:05.035786 29916 raft_consensus.cc:482] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:05.036266 29916 raft_consensus.cc:504] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.038174 29916 leader_election.cc:288] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:05.039660 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "696121417ea9403990456cfc6fc4f416" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:05.039218 29558 catalog_manager.cc:5484] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 0 to 1, leader changed from <none> to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 1 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:05.040302 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "696121417ea9403990456cfc6fc4f416" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:05.040774 29698 leader_election.cc:341] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
W20240501 20:01:05.041484 29700 leader_election.cc:341] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:05.041857 29700 leader_election.cc:302] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:05.042445 29916 raft_consensus.cc:2726] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:05.044319 29852 consensus_queue.cc:260] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.046790 29851 tablet_bootstrap.cc:654] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:05.048566 29933 consensus_queue.cc:1035] T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:01:05.048884 29852 ts_tablet_manager.cc:1398] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20240501 20:01:05.050804 29852 tablet_bootstrap.cc:492] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:05.052217 29849 tablet_bootstrap.cc:492] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:05.052650 29849 ts_tablet_manager.cc:1367] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.224s user 0.013s sys 0.000s
I20240501 20:01:05.055083 29851 tablet_bootstrap.cc:492] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:05.055155 29849 raft_consensus.cc:348] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.055611 29851 ts_tablet_manager.cc:1367] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.032s user 0.007s sys 0.002s
I20240501 20:01:05.056768 29852 tablet_bootstrap.cc:654] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:05.057336 29849 raft_consensus.cc:374] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:05.057664 29849 raft_consensus.cc:729] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:05.058585 29851 raft_consensus.cc:348] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.059337 29851 raft_consensus.cc:374] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:05.059605 29851 raft_consensus.cc:729] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:05.058367 29849 consensus_queue.cc:260] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.060191 29851 consensus_queue.cc:260] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.061890 29851 ts_tablet_manager.cc:1398] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.005s user 0.002s sys 0.002s
I20240501 20:01:05.062686 29851 tablet_bootstrap.cc:492] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:05.067394 29851 tablet_bootstrap.cc:654] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
W20240501 20:01:05.070415 29624 consensus_peers.cc:485] T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f -> Peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Couldn't send request to peer fdfeac4190e045f290e003b2298d988d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:05.073029 29849 ts_tablet_manager.cc:1398] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20240501 20:01:05.073972 29849 tablet_bootstrap.cc:492] T 8bd5e83faaff4e2da4e4cc22ba410d33 P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:05.079298 29849 tablet_bootstrap.cc:654] T 8bd5e83faaff4e2da4e4cc22ba410d33 P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:05.080971 29937 raft_consensus.cc:482] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:05.081703 29937 raft_consensus.cc:504] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.082617 29852 tablet_bootstrap.cc:492] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:05.083220 29852 ts_tablet_manager.cc:1367] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.033s user 0.007s sys 0.007s
I20240501 20:01:05.085736 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49da4246f5f6438abe0e7e8c9a523d80" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:05.085963 29937 leader_election.cc:288] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:05.086608 29733 raft_consensus.cc:2379] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 1: Already voted for candidate ee0d97247d394a8facfa774f15f3a67d in this term.
I20240501 20:01:05.087064 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49da4246f5f6438abe0e7e8c9a523d80" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:05.088515 29625 leader_election.cc:302] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:05.088639 29852 raft_consensus.cc:348] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.089325 29852 raft_consensus.cc:374] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:05.089615 29852 raft_consensus.cc:729] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:05.089620 29937 raft_consensus.cc:3037] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:05.090454 29852 consensus_queue.cc:260] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.092576 29852 ts_tablet_manager.cc:1398] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.009s user 0.006s sys 0.002s
I20240501 20:01:05.093441 29852 tablet_bootstrap.cc:492] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:05.098018 29937 raft_consensus.cc:2726] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:05.098659 29852 tablet_bootstrap.cc:654] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:05.104625 29849 tablet_bootstrap.cc:492] T 8bd5e83faaff4e2da4e4cc22ba410d33 P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:05.105358 29849 ts_tablet_manager.cc:1367] T 8bd5e83faaff4e2da4e4cc22ba410d33 P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.032s user 0.011s sys 0.006s
I20240501 20:01:05.115458 29853 consensus_queue.cc:1035] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:05.119000 29849 raft_consensus.cc:348] T 8bd5e83faaff4e2da4e4cc22ba410d33 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.120007 29849 raft_consensus.cc:374] T 8bd5e83faaff4e2da4e4cc22ba410d33 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:05.120378 29849 raft_consensus.cc:729] T 8bd5e83faaff4e2da4e4cc22ba410d33 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:05.121850 29849 consensus_queue.cc:260] T 8bd5e83faaff4e2da4e4cc22ba410d33 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.125710 29849 ts_tablet_manager.cc:1398] T 8bd5e83faaff4e2da4e4cc22ba410d33 P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.020s user 0.009s sys 0.000s
I20240501 20:01:05.126363 29851 tablet_bootstrap.cc:492] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:05.126837 29851 ts_tablet_manager.cc:1367] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.064s user 0.037s sys 0.013s
I20240501 20:01:05.126928 29849 tablet_bootstrap.cc:492] T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:05.129480 29851 raft_consensus.cc:348] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.130218 29851 raft_consensus.cc:374] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:05.130491 29851 raft_consensus.cc:729] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:05.131181 29851 consensus_queue.cc:260] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.133575 29851 ts_tablet_manager.cc:1398] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.006s user 0.006s sys 0.002s
I20240501 20:01:05.134505 29851 tablet_bootstrap.cc:492] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:05.135221 29852 tablet_bootstrap.cc:492] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:05.135648 29852 ts_tablet_manager.cc:1367] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.042s user 0.023s sys 0.015s
I20240501 20:01:05.136878 29849 tablet_bootstrap.cc:654] T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:05.138549 29852 raft_consensus.cc:348] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.139282 29852 raft_consensus.cc:374] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:05.139547 29852 raft_consensus.cc:729] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:05.140314 29852 consensus_queue.cc:260] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.142927 29852 ts_tablet_manager.cc:1398] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240501 20:01:05.144500 29852 tablet_bootstrap.cc:492] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:05.146567 29851 tablet_bootstrap.cc:654] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:05.147083 29916 consensus_queue.cc:1035] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:05.151882 29852 tablet_bootstrap.cc:654] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:05.161676 29853 raft_consensus.cc:482] T 8bd5e83faaff4e2da4e4cc22ba410d33 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:05.162434 29853 raft_consensus.cc:504] T 8bd5e83faaff4e2da4e4cc22ba410d33 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.164564 29853 leader_election.cc:288] T 8bd5e83faaff4e2da4e4cc22ba410d33 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:05.165525 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd5e83faaff4e2da4e4cc22ba410d33" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
W20240501 20:01:05.166913 29772 leader_election.cc:341] T 8bd5e83faaff4e2da4e4cc22ba410d33 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:05.167189 29852 tablet_bootstrap.cc:492] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:05.167270 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd5e83faaff4e2da4e4cc22ba410d33" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:05.167724 29852 ts_tablet_manager.cc:1367] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.009s
I20240501 20:01:05.167989 29733 raft_consensus.cc:2454] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 0.
I20240501 20:01:05.169137 29773 leader_election.cc:302] T 8bd5e83faaff4e2da4e4cc22ba410d33 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:05.171161 29896 raft_consensus.cc:2781] T 8bd5e83faaff4e2da4e4cc22ba410d33 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:05.171509 29896 raft_consensus.cc:482] T 8bd5e83faaff4e2da4e4cc22ba410d33 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:05.171800 29896 raft_consensus.cc:3037] T 8bd5e83faaff4e2da4e4cc22ba410d33 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:05.175555 29909 raft_consensus.cc:482] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:05.176030 29909 raft_consensus.cc:504] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.177985 29909 leader_election.cc:288] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:05.179001 29852 raft_consensus.cc:348] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.179419 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac2171683884f6f97cdfeb0a656cb49" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:05.179714 29852 raft_consensus.cc:374] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:05.180001 29852 raft_consensus.cc:729] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:05.180804 29852 consensus_queue.cc:260] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.181697 29660 raft_consensus.cc:2454] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 0.
I20240501 20:01:05.183671 29698 leader_election.cc:302] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:05.184509 29852 ts_tablet_manager.cc:1398] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.016s user 0.008s sys 0.000s
I20240501 20:01:05.185387 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac2171683884f6f97cdfeb0a656cb49" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:05.186040 29808 raft_consensus.cc:2454] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 0.
I20240501 20:01:05.186581 29914 raft_consensus.cc:2781] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:05.186314 29896 raft_consensus.cc:504] T 8bd5e83faaff4e2da4e4cc22ba410d33 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.186957 29914 raft_consensus.cc:482] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:05.186087 29856 consensus_queue.cc:1035] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:01:05.187383 29852 tablet_bootstrap.cc:492] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:05.187384 29914 raft_consensus.cc:3037] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:05.189157 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd5e83faaff4e2da4e4cc22ba410d33" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
W20240501 20:01:05.190960 29772 leader_election.cc:341] T 8bd5e83faaff4e2da4e4cc22ba410d33 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:05.192721 29896 leader_election.cc:288] T 8bd5e83faaff4e2da4e4cc22ba410d33 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:05.192955 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd5e83faaff4e2da4e4cc22ba410d33" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:05.193559 29732 raft_consensus.cc:3037] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:05.195174 29892 consensus_queue.cc:1035] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:05.199502 29914 raft_consensus.cc:504] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.200835 29909 raft_consensus.cc:482] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:05.201280 29909 raft_consensus.cc:504] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.202411 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac2171683884f6f97cdfeb0a656cb49" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:05.202972 29660 raft_consensus.cc:3037] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:05.203171 29909 leader_election.cc:288] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:05.200222 29732 raft_consensus.cc:2454] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
I20240501 20:01:05.205343 29852 tablet_bootstrap.cc:654] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:05.205421 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e38a2e91a3d48d09a22abce723af833" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:05.206895 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e38a2e91a3d48d09a22abce723af833" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
W20240501 20:01:05.207110 29700 leader_election.cc:341] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:05.207614 29658 raft_consensus.cc:2454] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 0.
I20240501 20:01:05.208958 29773 leader_election.cc:302] T 8bd5e83faaff4e2da4e4cc22ba410d33 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:05.210003 29660 raft_consensus.cc:2454] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
I20240501 20:01:05.210397 29892 raft_consensus.cc:2781] T 8bd5e83faaff4e2da4e4cc22ba410d33 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:05.211038 29892 raft_consensus.cc:686] T 8bd5e83faaff4e2da4e4cc22ba410d33 P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:05.212146 29892 consensus_queue.cc:237] T 8bd5e83faaff4e2da4e4cc22ba410d33 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.218294 29698 leader_election.cc:302] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:05.218513 29909 consensus_queue.cc:1035] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:01:05.219445 29856 raft_consensus.cc:2781] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:05.219934 29856 raft_consensus.cc:482] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:05.220366 29856 raft_consensus.cc:3037] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:05.224864 29698 leader_election.cc:302] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:05.227370 29851 tablet_bootstrap.cc:492] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:05.227983 29851 ts_tablet_manager.cc:1367] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.094s user 0.015s sys 0.007s
I20240501 20:01:05.228369 29852 tablet_bootstrap.cc:492] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:05.228447 29909 raft_consensus.cc:2781] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:05.228991 29852 ts_tablet_manager.cc:1367] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.042s user 0.012s sys 0.005s
I20240501 20:01:05.229300 29909 raft_consensus.cc:686] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:05.230180 29909 consensus_queue.cc:237] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.231525 29851 raft_consensus.cc:348] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.232354 29851 raft_consensus.cc:374] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:05.232651 29851 raft_consensus.cc:729] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:05.246912 29851 consensus_queue.cc:260] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.248288 29849 tablet_bootstrap.cc:492] T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:05.252444 29916 consensus_queue.cc:1035] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:05.249621 29851 ts_tablet_manager.cc:1398] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.021s user 0.002s sys 0.008s
I20240501 20:01:05.254490 29851 tablet_bootstrap.cc:492] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:05.247710 29852 raft_consensus.cc:348] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.260715 29852 raft_consensus.cc:374] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:05.261025 29852 raft_consensus.cc:729] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:05.261675 29852 consensus_queue.cc:260] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.263708 29852 ts_tablet_manager.cc:1398] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.034s user 0.006s sys 0.003s
I20240501 20:01:05.252785 29849 ts_tablet_manager.cc:1367] T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.126s user 0.016s sys 0.004s
I20240501 20:01:05.265841 29851 tablet_bootstrap.cc:654] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:05.266327 29849 raft_consensus.cc:348] T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.267057 29849 raft_consensus.cc:374] T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:05.267314 29849 raft_consensus.cc:729] T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:05.272361 29914 leader_election.cc:288] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:05.273675 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac2171683884f6f97cdfeb0a656cb49" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:05.274348 29852 tablet_bootstrap.cc:492] T 696121417ea9403990456cfc6fc4f416 P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:05.274487 29808 raft_consensus.cc:3037] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:05.275276 29909 consensus_queue.cc:1035] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:05.271565 29849 consensus_queue.cc:260] T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.277832 29849 ts_tablet_manager.cc:1398] T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.014s user 0.004s sys 0.000s
I20240501 20:01:05.280592 29851 tablet_bootstrap.cc:492] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:05.281055 29851 ts_tablet_manager.cc:1367] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.009s
I20240501 20:01:05.282737 29852 tablet_bootstrap.cc:654] T 696121417ea9403990456cfc6fc4f416 P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:05.283370 29851 raft_consensus.cc:348] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.284006 29851 raft_consensus.cc:374] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:05.284309 29851 raft_consensus.cc:729] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:05.284565 29849 tablet_bootstrap.cc:492] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:05.250170 29856 raft_consensus.cc:504] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.286892 29851 consensus_queue.cc:260] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.288960 29851 ts_tablet_manager.cc:1398] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240501 20:01:05.288771 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e38a2e91a3d48d09a22abce723af833" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:05.289392 29658 raft_consensus.cc:3037] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:05.289525 29914 raft_consensus.cc:482] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:05.290210 29851 tablet_bootstrap.cc:492] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:05.290287 29914 raft_consensus.cc:504] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.294622 29914 leader_election.cc:288] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:05.297183 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be383be783234855878cb8fd442c7b05" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:05.342905 29558 catalog_manager.cc:5484] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 0 to 1, leader changed from <none> to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 1 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:05.298331 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be383be783234855878cb8fd442c7b05" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:05.298048 29856 leader_election.cc:288] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:05.347419 29700 leader_election.cc:341] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:05.297358 29808 raft_consensus.cc:2454] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
I20240501 20:01:05.306645 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e38a2e91a3d48d09a22abce723af833" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:05.324280 29659 raft_consensus.cc:3037] T cb05df43bdfa479b902b1da297fb5974 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
W20240501 20:01:05.351189 29698 leader_election.cc:341] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
W20240501 20:01:05.351646 29700 leader_election.cc:341] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:05.351724 29698 leader_election.cc:302] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:05.355376 29849 tablet_bootstrap.cc:654] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:05.308005 29658 raft_consensus.cc:2454] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
I20240501 20:01:05.356849 29914 raft_consensus.cc:2726] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:05.359131 29852 tablet_bootstrap.cc:492] T 696121417ea9403990456cfc6fc4f416 P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:05.358733 29556 catalog_manager.cc:5484] T 8bd5e83faaff4e2da4e4cc22ba410d33 P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 0 to 1, leader changed from <none> to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 1 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:05.360284 29852 ts_tablet_manager.cc:1367] T 696121417ea9403990456cfc6fc4f416 P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.086s user 0.008s sys 0.005s
I20240501 20:01:05.360801 29937 raft_consensus.cc:482] T cb05df43bdfa479b902b1da297fb5974 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:05.363312 29937 raft_consensus.cc:504] T cb05df43bdfa479b902b1da297fb5974 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.368624 29851 tablet_bootstrap.cc:654] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:05.369244 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05df43bdfa479b902b1da297fb5974" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:05.372386 29937 leader_election.cc:288] T cb05df43bdfa479b902b1da297fb5974 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:05.374111 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05df43bdfa479b902b1da297fb5974" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:05.375715 29624 leader_election.cc:302] T cb05df43bdfa479b902b1da297fb5974 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
W20240501 20:01:05.356659 29960 log.cc:924] Time spent T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d: Append to log took a long time: real 0.062s user 0.001s sys 0.000s
I20240501 20:01:05.360924 29698 leader_election.cc:302] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:05.381918 29856 raft_consensus.cc:2781] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:05.382424 29856 raft_consensus.cc:686] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:05.378203 29851 tablet_bootstrap.cc:492] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:05.383062 29851 ts_tablet_manager.cc:1367] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.093s user 0.015s sys 0.004s
I20240501 20:01:05.383231 29856 consensus_queue.cc:237] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.384903 29807 raft_consensus.cc:3037] T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:05.385615 29851 raft_consensus.cc:348] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.386345 29851 raft_consensus.cc:374] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:05.386668 29851 raft_consensus.cc:729] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:05.387285 29851 consensus_queue.cc:260] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.388058 29914 consensus_queue.cc:1035] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20240501 20:01:05.398689 29852 raft_consensus.cc:348] T 696121417ea9403990456cfc6fc4f416 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.399890 29852 raft_consensus.cc:374] T 696121417ea9403990456cfc6fc4f416 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:05.400245 29852 raft_consensus.cc:729] T 696121417ea9403990456cfc6fc4f416 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:05.401093 29852 consensus_queue.cc:260] T 696121417ea9403990456cfc6fc4f416 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.389393 29851 ts_tablet_manager.cc:1398] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240501 20:01:05.413115 29849 tablet_bootstrap.cc:492] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:05.413287 29851 tablet_bootstrap.cc:492] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:05.413578 29849 ts_tablet_manager.cc:1367] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.129s user 0.035s sys 0.049s
I20240501 20:01:05.417377 29879 consensus_queue.cc:1035] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:01:05.417604 29849 raft_consensus.cc:348] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.418455 29849 raft_consensus.cc:374] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:05.418882 29849 raft_consensus.cc:729] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:05.419587 29849 consensus_queue.cc:260] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.422276 29849 ts_tablet_manager.cc:1398] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20240501 20:01:05.423177 29849 tablet_bootstrap.cc:492] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:05.424397 29914 consensus_queue.cc:1035] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:01:05.425513 29851 tablet_bootstrap.cc:654] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:05.433902 29852 ts_tablet_manager.cc:1398] T 696121417ea9403990456cfc6fc4f416 P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.060s user 0.012s sys 0.000s
I20240501 20:01:05.434553 29849 tablet_bootstrap.cc:654] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:05.434958 29852 tablet_bootstrap.cc:492] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:05.434859 29856 raft_consensus.cc:482] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:05.435374 29856 raft_consensus.cc:504] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.438334 29933 raft_consensus.cc:482] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:05.438517 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f261d2a05bf4c838532049601694b4a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:05.439198 29660 raft_consensus.cc:2454] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 0.
I20240501 20:01:05.438867 29933 raft_consensus.cc:504] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.441306 29933 leader_election.cc:288] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:05.442327 29558 catalog_manager.cc:5484] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 0 to 1, leader changed from <none> to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 1 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:05.443210 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48630ae6eb394ac1b54955dd01cf0ed2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:05.443641 29852 tablet_bootstrap.cc:654] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
W20240501 20:01:05.445405 29624 leader_election.cc:341] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:05.446765 29698 leader_election.cc:302] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:05.447671 29914 raft_consensus.cc:2781] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:05.448117 29914 raft_consensus.cc:482] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:05.448526 29914 raft_consensus.cc:3037] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:05.449030 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48630ae6eb394ac1b54955dd01cf0ed2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:05.450384 29625 leader_election.cc:341] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:05.450897 29625 leader_election.cc:302] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:05.451632 29933 raft_consensus.cc:2726] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:05.456375 29914 raft_consensus.cc:504] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.458633 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f261d2a05bf4c838532049601694b4a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:05.463620 29700 leader_election.cc:341] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:05.459300 29856 leader_election.cc:288] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:05.471379 29849 tablet_bootstrap.cc:492] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:05.471884 29933 consensus_queue.cc:1035] T 1ca3948261374cef8b1b206a43c00173 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:05.472278 29852 tablet_bootstrap.cc:492] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:05.472916 29852 ts_tablet_manager.cc:1367] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.038s user 0.013s sys 0.000s
I20240501 20:01:05.472754 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f261d2a05bf4c838532049601694b4a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:05.473416 29660 raft_consensus.cc:3037] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:05.475644 29851 tablet_bootstrap.cc:492] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:05.475500 29852 raft_consensus.cc:348] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.476120 29851 ts_tablet_manager.cc:1367] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.063s user 0.012s sys 0.000s
I20240501 20:01:05.476275 29852 raft_consensus.cc:374] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:05.476598 29852 raft_consensus.cc:729] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:05.477018 29887 consensus_queue.cc:1035] T 1ca3948261374cef8b1b206a43c00173 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:05.477175 29852 consensus_queue.cc:260] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.478492 29851 raft_consensus.cc:348] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.479185 29851 raft_consensus.cc:374] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:05.479452 29851 raft_consensus.cc:729] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:05.471863 29849 ts_tablet_manager.cc:1367] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.049s user 0.009s sys 0.003s
I20240501 20:01:05.482053 29851 consensus_queue.cc:260] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.486577 29852 ts_tablet_manager.cc:1398] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20240501 20:01:05.487789 29852 tablet_bootstrap.cc:492] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:05.484867 29660 raft_consensus.cc:2454] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
I20240501 20:01:05.492009 29914 leader_election.cc:288] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:05.482020 29883 raft_consensus.cc:2726] T cb05df43bdfa479b902b1da297fb5974 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:05.493369 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f261d2a05bf4c838532049601694b4a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:05.492583 29914 raft_consensus.cc:482] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:05.495586 29698 leader_election.cc:302] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:05.496438 29856 raft_consensus.cc:2781] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:05.496887 29856 raft_consensus.cc:686] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:05.497339 29849 raft_consensus.cc:348] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.498060 29849 raft_consensus.cc:374] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:05.497658 29856 consensus_queue.cc:237] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.498337 29849 raft_consensus.cc:729] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:05.501438 29851 ts_tablet_manager.cc:1398] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.025s user 0.006s sys 0.000s
I20240501 20:01:05.502342 29851 tablet_bootstrap.cc:492] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:05.496429 29914 raft_consensus.cc:504] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.571411 29914 leader_election.cc:288] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:05.597666 29849 consensus_queue.cc:260] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.599416 29808 raft_consensus.cc:3037] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:05.601572 29849 ts_tablet_manager.cc:1398] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.120s user 0.007s sys 0.001s
I20240501 20:01:05.602522 29849 tablet_bootstrap.cc:492] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:05.610096 29808 raft_consensus.cc:2454] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
I20240501 20:01:05.611238 29852 tablet_bootstrap.cc:654] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:05.625507 29849 tablet_bootstrap.cc:654] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:05.628175 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d86f8848a184a87947fd927c44e5a35" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:05.630084 29808 raft_consensus.cc:2454] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 0.
I20240501 20:01:05.632618 29700 leader_election.cc:302] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:05.634047 29856 raft_consensus.cc:2781] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:05.634570 29856 raft_consensus.cc:482] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:05.634982 29856 raft_consensus.cc:3037] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:05.662781 29852 tablet_bootstrap.cc:492] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:05.663280 29852 ts_tablet_manager.cc:1367] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.176s user 0.061s sys 0.058s
I20240501 20:01:05.641010 29896 raft_consensus.cc:482] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:05.664579 29896 raft_consensus.cc:504] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.665990 29660 raft_consensus.cc:3037] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:05.666618 29896 leader_election.cc:288] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:05.662597 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d86f8848a184a87947fd927c44e5a35" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:05.679242 29851 tablet_bootstrap.cc:654] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:05.674369 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "696121417ea9403990456cfc6fc4f416" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:05.698020 29734 raft_consensus.cc:2454] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 0.
I20240501 20:01:05.702993 29856 raft_consensus.cc:504] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.709785 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "696121417ea9403990456cfc6fc4f416" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:05.711331 29732 raft_consensus.cc:3037] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:05.785406 29659 raft_consensus.cc:2454] T 696121417ea9403990456cfc6fc4f416 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 0.
I20240501 20:01:05.723080 29914 raft_consensus.cc:482] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:05.786732 29916 consensus_queue.cc:1035] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:05.790043 29856 leader_election.cc:288] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:05.790802 29933 raft_consensus.cc:482] T 696121417ea9403990456cfc6fc4f416 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:05.791270 29933 raft_consensus.cc:504] T 696121417ea9403990456cfc6fc4f416 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.793012 29933 leader_election.cc:288] T 696121417ea9403990456cfc6fc4f416 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:05.794010 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "696121417ea9403990456cfc6fc4f416" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:05.794661 29806 raft_consensus.cc:2454] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 0.
I20240501 20:01:05.795253 29772 leader_election.cc:302] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:05.796340 29853 raft_consensus.cc:2781] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:05.796304 29625 leader_election.cc:302] T 696121417ea9403990456cfc6fc4f416 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:05.796834 29853 raft_consensus.cc:482] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:05.797235 29853 raft_consensus.cc:3037] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:05.798014 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d86f8848a184a87947fd927c44e5a35" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:05.798552 29808 raft_consensus.cc:3037] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:05.780066 29849 tablet_bootstrap.cc:492] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:05.777768 29852 raft_consensus.cc:348] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.719174 29892 consensus_queue.cc:1035] T 8bd5e83faaff4e2da4e4cc22ba410d33 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.063s
W20240501 20:01:05.687916 29698 leader_election.cc:341] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:05.717855 29887 raft_consensus.cc:482] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:05.799427 29852 raft_consensus.cc:374] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:05.805881 29852 raft_consensus.cc:729] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:05.806432 29852 consensus_queue.cc:260] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.799800 29849 ts_tablet_manager.cc:1367] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.197s user 0.028s sys 0.056s
I20240501 20:01:05.804386 29851 tablet_bootstrap.cc:492] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:05.807437 29851 ts_tablet_manager.cc:1367] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.305s user 0.021s sys 0.007s
I20240501 20:01:05.810292 29852 ts_tablet_manager.cc:1398] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.147s user 0.005s sys 0.001s
I20240501 20:01:05.786846 29914 raft_consensus.cc:504] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.814087 29851 raft_consensus.cc:348] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.811187 29852 tablet_bootstrap.cc:492] T 2d86f8848a184a87947fd927c44e5a35 P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:05.815480 29851 raft_consensus.cc:374] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:05.808876 29887 raft_consensus.cc:504] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.815861 29851 raft_consensus.cc:729] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
W20240501 20:01:05.820526 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 135159us
I20240501 20:01:05.824913 29914 leader_election.cc:288] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:05.826187 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649b71f9f267437ca51c758dd9ca6987" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:05.827093 29853 raft_consensus.cc:504] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:05.827636 29700 leader_election.cc:341] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:05.804952 29979 raft_consensus.cc:2781] T 696121417ea9403990456cfc6fc4f416 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:05.831161 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "696121417ea9403990456cfc6fc4f416" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:05.831063 29851 consensus_queue.cc:260] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.837242 29808 raft_consensus.cc:2454] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
I20240501 20:01:05.839682 29700 leader_election.cc:302] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:05.840181 29851 ts_tablet_manager.cc:1398] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.032s user 0.009s sys 0.000s
I20240501 20:01:05.821919 29849 raft_consensus.cc:348] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.840732 29856 raft_consensus.cc:2781] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:05.840941 29849 raft_consensus.cc:374] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:05.831295 29979 raft_consensus.cc:482] T 696121417ea9403990456cfc6fc4f416 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:05.833166 29853 leader_election.cc:288] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:05.841344 29849 raft_consensus.cc:729] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:05.845305 29849 consensus_queue.cc:260] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.847352 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "696121417ea9403990456cfc6fc4f416" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:05.847293 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d86f8848a184a87947fd927c44e5a35" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:05.841428 29856 raft_consensus.cc:686] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:05.849749 29856 consensus_queue.cc:237] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.864420 29851 tablet_bootstrap.cc:492] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:05.843415 29733 raft_consensus.cc:2454] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 0.
I20240501 20:01:05.850998 29979 raft_consensus.cc:3037] T 696121417ea9403990456cfc6fc4f416 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:05.877131 29909 raft_consensus.cc:482] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:05.877707 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649b71f9f267437ca51c758dd9ca6987" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:05.928246 29977 raft_consensus.cc:482] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:05.930459 29909 raft_consensus.cc:504] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.933504 29909 leader_election.cc:288] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:05.913550 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "696121417ea9403990456cfc6fc4f416" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:05.936352 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd5e83faaff4e2da4e4cc22ba410d33" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:05.938670 29887 leader_election.cc:288] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:05.940038 29735 raft_consensus.cc:3037] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:05.928995 29977 raft_consensus.cc:504] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.957397 29849 ts_tablet_manager.cc:1398] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.150s user 0.004s sys 0.010s
I20240501 20:01:05.964455 29849 tablet_bootstrap.cc:492] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
W20240501 20:01:05.957650 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 44720.1us
I20240501 20:01:05.974472 29883 raft_consensus.cc:482] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:05.975064 29883 raft_consensus.cc:504] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:05.975165 29558 catalog_manager.cc:5484] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 0 to 1, leader changed from <none> to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 1 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:05.977651 29977 leader_election.cc:288] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:05.977850 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649b71f9f267437ca51c758dd9ca6987" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:05.979074 29625 leader_election.cc:341] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:05.979908 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d88139f89c47ff84cad674147aae0f" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:05.967413 29659 raft_consensus.cc:2454] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 0.
I20240501 20:01:05.984158 29852 tablet_bootstrap.cc:654] T 2d86f8848a184a87947fd927c44e5a35 P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:05.982142 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd5e83faaff4e2da4e4cc22ba410d33" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
W20240501 20:01:05.982108 29698 leader_election.cc:341] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:05.985464 29733 raft_consensus.cc:2454] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:05.996001 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be383be783234855878cb8fd442c7b05" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:05.996191 29916 consensus_queue.cc:1035] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:05.996714 29808 raft_consensus.cc:2454] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 0.
I20240501 20:01:06.000864 29625 leader_election.cc:302] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:06.001538 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649b71f9f267437ca51c758dd9ca6987" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:06.003083 29978 raft_consensus.cc:2781] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:06.003489 29733 raft_consensus.cc:2454] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 0.
I20240501 20:01:06.003561 29978 raft_consensus.cc:482] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:06.004457 29978 raft_consensus.cc:3037] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:06.003168 29624 leader_election.cc:302] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:06.006594 29984 consensus_queue.cc:1035] T 8bd5e83faaff4e2da4e4cc22ba410d33 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240501 20:01:05.993324 29883 leader_election.cc:288] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:06.009303 29624 leader_election.cc:302] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
W20240501 20:01:05.995960 29700 consensus_peers.cc:485] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d -> Peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Couldn't send request to peer ee0d97247d394a8facfa774f15f3a67d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:06.010787 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be383be783234855878cb8fd442c7b05" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:06.010010 29977 raft_consensus.cc:2781] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:06.018384 29887 raft_consensus.cc:2781] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:06.019923 29887 raft_consensus.cc:482] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:06.020174 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d88139f89c47ff84cad674147aae0f" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:06.021266 29977 raft_consensus.cc:482] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:06.021646 29977 raft_consensus.cc:3037] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:06.018378 29731 raft_consensus.cc:2454] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 0.
I20240501 20:01:06.024535 29887 raft_consensus.cc:3037] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:06.037635 29851 tablet_bootstrap.cc:654] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:06.042609 29849 tablet_bootstrap.cc:654] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:06.029469 29698 leader_election.cc:302] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:06.047101 29909 raft_consensus.cc:2781] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240501 20:01:06.047629 29983 log.cc:924] Time spent T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.158s user 0.001s sys 0.000s
I20240501 20:01:06.047634 29909 raft_consensus.cc:482] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:06.049821 29909 raft_consensus.cc:3037] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:06.050556 29735 raft_consensus.cc:2454] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
I20240501 20:01:06.069010 29887 raft_consensus.cc:504] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.076455 29887 leader_election.cc:288] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:06.077186 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649b71f9f267437ca51c758dd9ca6987" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:06.078541 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649b71f9f267437ca51c758dd9ca6987" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
W20240501 20:01:06.079763 29625 leader_election.cc:341] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:06.082041 29852 tablet_bootstrap.cc:492] T 2d86f8848a184a87947fd927c44e5a35 P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:06.082459 29852 ts_tablet_manager.cc:1367] T 2d86f8848a184a87947fd927c44e5a35 P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.271s user 0.015s sys 0.023s
I20240501 20:01:06.082278 29979 raft_consensus.cc:504] T 696121417ea9403990456cfc6fc4f416 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.083861 29660 raft_consensus.cc:2379] T 696121417ea9403990456cfc6fc4f416 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 1: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:06.084597 29852 raft_consensus.cc:348] T 2d86f8848a184a87947fd927c44e5a35 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.085319 29852 raft_consensus.cc:374] T 2d86f8848a184a87947fd927c44e5a35 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:06.085556 29852 raft_consensus.cc:729] T 2d86f8848a184a87947fd927c44e5a35 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:06.086159 29852 consensus_queue.cc:260] T 2d86f8848a184a87947fd927c44e5a35 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.086719 29978 raft_consensus.cc:504] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.087760 29852 ts_tablet_manager.cc:1398] T 2d86f8848a184a87947fd927c44e5a35 P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:01:06.088624 29852 tablet_bootstrap.cc:492] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:06.088665 29773 leader_election.cc:302] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:06.090344 29985 raft_consensus.cc:2781] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:06.092315 29985 raft_consensus.cc:686] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:06.093911 29856 raft_consensus.cc:482] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:06.094983 29856 raft_consensus.cc:504] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.094882 29985 consensus_queue.cc:237] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.095650 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "696121417ea9403990456cfc6fc4f416" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:06.096736 29735 raft_consensus.cc:2379] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 1: Already voted for candidate ee0d97247d394a8facfa774f15f3a67d in this term.
I20240501 20:01:06.096542 29989 consensus_queue.cc:1035] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:01:06.097611 29856 leader_election.cc:288] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:06.098460 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "456035a1cb3f4f7da52d0fc1dd1c656c" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:06.099750 29700 leader_election.cc:341] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:06.100773 29698 leader_election.cc:302] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:06.101384 29856 consensus_queue.cc:1035] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:06.104903 29989 raft_consensus.cc:2726] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:06.105461 29989 consensus_queue.cc:1035] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:06.105875 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "456035a1cb3f4f7da52d0fc1dd1c656c" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
W20240501 20:01:06.111562 29698 leader_election.cc:341] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:06.114568 29698 leader_election.cc:302] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:06.115350 29914 raft_consensus.cc:2726] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:06.124248 29851 tablet_bootstrap.cc:492] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:06.124008 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "696121417ea9403990456cfc6fc4f416" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:06.124727 29851 ts_tablet_manager.cc:1367] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.267s user 0.016s sys 0.003s
I20240501 20:01:06.126147 29852 tablet_bootstrap.cc:654] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:06.116209 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be383be783234855878cb8fd442c7b05" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:06.112418 29979 leader_election.cc:288] T 696121417ea9403990456cfc6fc4f416 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:06.127804 29851 raft_consensus.cc:348] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.132576 29851 raft_consensus.cc:374] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:06.130039 29625 leader_election.cc:302] T 696121417ea9403990456cfc6fc4f416 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:06.130066 29735 raft_consensus.cc:3037] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:06.131314 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be383be783234855878cb8fd442c7b05" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:06.136790 29807 raft_consensus.cc:3037] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:06.137054 29977 raft_consensus.cc:504] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.138113 29658 raft_consensus.cc:1226] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:06.141175 29977 leader_election.cc:288] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:06.141763 29977 raft_consensus.cc:2726] T 696121417ea9403990456cfc6fc4f416 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 20:01:06.142264 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd5e83faaff4e2da4e4cc22ba410d33" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:06.122510 29978 leader_election.cc:288] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:06.143532 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd5e83faaff4e2da4e4cc22ba410d33" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:06.144754 29851 raft_consensus.cc:729] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:06.150008 29625 leader_election.cc:302] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:06.151844 29977 raft_consensus.cc:2726] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 20:01:06.116304 29909 raft_consensus.cc:504] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:06.154280 29606 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240501 20:01:06.152937 29851 consensus_queue.cc:260] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.143868 29985 consensus_queue.cc:1046] T 8bd5e83faaff4e2da4e4cc22ba410d33 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:06.158131 29851 ts_tablet_manager.cc:1398] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.033s user 0.006s sys 0.000s
I20240501 20:01:06.158759 29807 raft_consensus.cc:2454] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:06.159332 29851 tablet_bootstrap.cc:492] T 010ceabf3e5a4b1c9330b80dade765bb P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:06.160570 29625 leader_election.cc:302] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:06.161751 29978 raft_consensus.cc:2781] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:06.162377 29978 raft_consensus.cc:686] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:06.162981 29732 raft_consensus.cc:2379] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 1: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:06.163493 29978 consensus_queue.cc:237] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.163496 29896 raft_consensus.cc:3032] T 8bd5e83faaff4e2da4e4cc22ba410d33 P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:06.165926 29896 raft_consensus.cc:729] T 8bd5e83faaff4e2da4e4cc22ba410d33 P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:06.166605 29624 leader_election.cc:302] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:06.170817 29909 leader_election.cc:288] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:06.175810 29849 tablet_bootstrap.cc:492] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:06.176293 29849 ts_tablet_manager.cc:1367] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.212s user 0.006s sys 0.014s
I20240501 20:01:06.176920 29979 raft_consensus.cc:2726] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 20:01:06.182590 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649b71f9f267437ca51c758dd9ca6987" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:06.182257 29896 consensus_queue.cc:260] T 8bd5e83faaff4e2da4e4cc22ba410d33 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.183972 29896 raft_consensus.cc:3037] T 8bd5e83faaff4e2da4e4cc22ba410d33 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:06.188768 29735 raft_consensus.cc:2454] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:06.181105 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649b71f9f267437ca51c758dd9ca6987" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:06.200364 29659 raft_consensus.cc:2379] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d in current term 1: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:06.203493 29849 raft_consensus.cc:348] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.204495 29849 raft_consensus.cc:374] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:06.204882 29849 raft_consensus.cc:729] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:06.205507 29849 consensus_queue.cc:260] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.207743 29849 ts_tablet_manager.cc:1398] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.031s user 0.005s sys 0.001s
I20240501 20:01:06.208138 29807 raft_consensus.cc:3037] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:06.208765 29849 tablet_bootstrap.cc:492] T 48630ae6eb394ac1b54955dd01cf0ed2 P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:06.216238 29849 tablet_bootstrap.cc:654] T 48630ae6eb394ac1b54955dd01cf0ed2 P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:06.221665 29851 tablet_bootstrap.cc:654] T 010ceabf3e5a4b1c9330b80dade765bb P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:06.222505 29807 raft_consensus.cc:2454] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
I20240501 20:01:06.224534 29700 leader_election.cc:302] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:06.229763 29849 tablet_bootstrap.cc:492] T 48630ae6eb394ac1b54955dd01cf0ed2 P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:06.230479 29849 ts_tablet_manager.cc:1367] T 48630ae6eb394ac1b54955dd01cf0ed2 P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.011s
I20240501 20:01:06.230865 29989 raft_consensus.cc:2781] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:06.233808 29849 raft_consensus.cc:348] T 48630ae6eb394ac1b54955dd01cf0ed2 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.234735 29849 raft_consensus.cc:374] T 48630ae6eb394ac1b54955dd01cf0ed2 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:06.235077 29849 raft_consensus.cc:729] T 48630ae6eb394ac1b54955dd01cf0ed2 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:06.235883 29849 consensus_queue.cc:260] T 48630ae6eb394ac1b54955dd01cf0ed2 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.238205 29989 raft_consensus.cc:686] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:06.238651 29849 ts_tablet_manager.cc:1398] T 48630ae6eb394ac1b54955dd01cf0ed2 P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240501 20:01:06.240260 29849 tablet_bootstrap.cc:492] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:06.239095 29989 consensus_queue.cc:237] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.242942 29978 raft_consensus.cc:482] T 2d86f8848a184a87947fd927c44e5a35 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:06.243573 29978 raft_consensus.cc:504] T 2d86f8848a184a87947fd927c44e5a35 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.247841 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d86f8848a184a87947fd927c44e5a35" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:06.248001 29978 leader_election.cc:288] T 2d86f8848a184a87947fd927c44e5a35 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:06.249114 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d86f8848a184a87947fd927c44e5a35" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:06.249811 29808 raft_consensus.cc:2379] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 1: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:06.250795 29625 leader_election.cc:302] T 2d86f8848a184a87947fd927c44e5a35 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:06.251389 29849 tablet_bootstrap.cc:654] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:06.251943 29978 raft_consensus.cc:3037] T 2d86f8848a184a87947fd927c44e5a35 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:06.252341 29852 tablet_bootstrap.cc:492] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:06.252800 29852 ts_tablet_manager.cc:1367] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.164s user 0.021s sys 0.025s
I20240501 20:01:06.258455 29852 raft_consensus.cc:348] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.259207 29852 raft_consensus.cc:374] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:06.259464 29852 raft_consensus.cc:729] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:06.260339 29851 tablet_bootstrap.cc:492] T 010ceabf3e5a4b1c9330b80dade765bb P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:06.260782 29851 ts_tablet_manager.cc:1367] T 010ceabf3e5a4b1c9330b80dade765bb P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.102s user 0.016s sys 0.027s
I20240501 20:01:06.254406 29556 catalog_manager.cc:5484] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 0 to 1, leader changed from <none> to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 1 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:06.260840 29852 consensus_queue.cc:260] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.263103 29852 ts_tablet_manager.cc:1398] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.010s user 0.005s sys 0.002s
I20240501 20:01:06.264173 29852 tablet_bootstrap.cc:492] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:06.264940 29978 raft_consensus.cc:2726] T 2d86f8848a184a87947fd927c44e5a35 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:06.265231 29849 tablet_bootstrap.cc:492] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:06.265683 29849 ts_tablet_manager.cc:1367] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.026s user 0.013s sys 0.000s
I20240501 20:01:06.263144 29851 raft_consensus.cc:348] T 010ceabf3e5a4b1c9330b80dade765bb P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.266863 29851 raft_consensus.cc:374] T 010ceabf3e5a4b1c9330b80dade765bb P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:06.267210 29851 raft_consensus.cc:729] T 010ceabf3e5a4b1c9330b80dade765bb P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:06.267927 29851 consensus_queue.cc:260] T 010ceabf3e5a4b1c9330b80dade765bb P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.269729 29849 raft_consensus.cc:348] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.270282 29849 raft_consensus.cc:374] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:06.270486 29849 raft_consensus.cc:729] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:06.271015 29849 consensus_queue.cc:260] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.272548 29851 ts_tablet_manager.cc:1398] T 010ceabf3e5a4b1c9330b80dade765bb P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20240501 20:01:06.272850 29849 ts_tablet_manager.cc:1398] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240501 20:01:06.273531 29851 tablet_bootstrap.cc:492] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:06.273706 29849 tablet_bootstrap.cc:492] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:06.279096 29851 tablet_bootstrap.cc:654] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:06.280021 29554 catalog_manager.cc:5484] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 1 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:06.282693 29849 tablet_bootstrap.cc:654] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:06.284637 29852 tablet_bootstrap.cc:654] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:06.294993 29851 tablet_bootstrap.cc:492] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:06.295235 29852 tablet_bootstrap.cc:492] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:06.295452 29851 ts_tablet_manager.cc:1367] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.014s
I20240501 20:01:06.295696 29852 ts_tablet_manager.cc:1367] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.032s user 0.009s sys 0.004s
I20240501 20:01:06.296104 29849 tablet_bootstrap.cc:492] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:06.296587 29849 ts_tablet_manager.cc:1367] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.023s user 0.012s sys 0.005s
I20240501 20:01:06.298440 29851 raft_consensus.cc:348] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.299183 29851 raft_consensus.cc:374] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:06.298883 29849 raft_consensus.cc:348] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.299856 29849 raft_consensus.cc:374] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:06.300195 29849 raft_consensus.cc:729] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:06.300482 29852 raft_consensus.cc:348] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.301172 29852 raft_consensus.cc:374] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:06.300885 29849 consensus_queue.cc:260] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.301440 29852 raft_consensus.cc:729] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:06.299491 29851 raft_consensus.cc:729] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:06.301838 29558 catalog_manager.cc:5484] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 0 to 1, leader changed from <none> to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 1 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:06.303113 29851 consensus_queue.cc:260] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.303599 29849 ts_tablet_manager.cc:1398] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240501 20:01:06.302104 29852 consensus_queue.cc:260] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.305095 29849 tablet_bootstrap.cc:492] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:06.312374 29849 tablet_bootstrap.cc:654] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:06.313230 29852 ts_tablet_manager.cc:1398] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20240501 20:01:06.314263 29852 tablet_bootstrap.cc:492] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:06.316184 29851 ts_tablet_manager.cc:1398] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.020s user 0.004s sys 0.004s
I20240501 20:01:06.317121 29851 tablet_bootstrap.cc:492] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:06.318073 29849 tablet_bootstrap.cc:492] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:06.318487 29849 ts_tablet_manager.cc:1367] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20240501 20:01:06.319967 29852 tablet_bootstrap.cc:654] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:06.317951 29989 raft_consensus.cc:482] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:06.320808 29849 raft_consensus.cc:348] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.321743 29849 raft_consensus.cc:374] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:06.321414 29989 raft_consensus.cc:504] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.322044 29849 raft_consensus.cc:729] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:06.322777 29851 tablet_bootstrap.cc:654] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:06.322757 29849 consensus_queue.cc:260] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.323349 29989 leader_election.cc:288] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:06.324818 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38cbeb0dc0c14e3d9182101eea885625" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:06.325124 29849 ts_tablet_manager.cc:1398] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240501 20:01:06.325438 29807 raft_consensus.cc:2454] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 0.
I20240501 20:01:06.326297 29849 tablet_bootstrap.cc:492] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:06.326076 29557 catalog_manager.cc:5484] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 0 to 1, leader changed from <none> to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 1 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:06.324812 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38cbeb0dc0c14e3d9182101eea885625" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
W20240501 20:01:06.328352 29698 leader_election.cc:341] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:06.331569 29700 leader_election.cc:302] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:06.332657 29989 raft_consensus.cc:2781] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:06.333019 29989 raft_consensus.cc:482] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:06.333343 29989 raft_consensus.cc:3037] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:06.337507 29849 tablet_bootstrap.cc:654] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:06.343055 29989 raft_consensus.cc:504] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.346571 29989 leader_election.cc:288] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:06.350204 29852 tablet_bootstrap.cc:492] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:06.350706 29852 ts_tablet_manager.cc:1367] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.037s user 0.011s sys 0.006s
I20240501 20:01:06.350780 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38cbeb0dc0c14e3d9182101eea885625" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
W20240501 20:01:06.352010 29698 leader_election.cc:341] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:06.352815 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38cbeb0dc0c14e3d9182101eea885625" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:06.353370 29807 raft_consensus.cc:3037] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:06.352988 29852 raft_consensus.cc:348] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.354686 29852 raft_consensus.cc:374] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:06.354979 29852 raft_consensus.cc:729] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:06.357185 29852 consensus_queue.cc:260] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.360318 29852 ts_tablet_manager.cc:1398] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20240501 20:01:06.360606 29807 raft_consensus.cc:2454] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
I20240501 20:01:06.361146 29852 tablet_bootstrap.cc:492] T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:06.361830 29700 leader_election.cc:302] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:06.362620 29849 tablet_bootstrap.cc:492] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:06.363037 29849 ts_tablet_manager.cc:1367] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.037s user 0.013s sys 0.000s
I20240501 20:01:06.364454 29989 raft_consensus.cc:2781] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:06.365409 29849 raft_consensus.cc:348] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.366158 29849 raft_consensus.cc:374] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:06.365942 29989 raft_consensus.cc:686] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:06.366677 29849 raft_consensus.cc:729] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:06.367453 29849 consensus_queue.cc:260] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.367563 29989 consensus_queue.cc:237] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.376855 29852 tablet_bootstrap.cc:654] T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:06.381098 29849 ts_tablet_manager.cc:1398] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20240501 20:01:06.382166 29849 tablet_bootstrap.cc:492] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:06.384708 29851 tablet_bootstrap.cc:492] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:06.385226 29851 ts_tablet_manager.cc:1367] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.068s user 0.008s sys 0.018s
I20240501 20:01:06.387791 29851 raft_consensus.cc:348] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.388484 29851 raft_consensus.cc:374] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:06.388809 29851 raft_consensus.cc:729] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:06.389461 29851 consensus_queue.cc:260] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.390523 29914 consensus_queue.cc:1035] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:06.393939 29851 ts_tablet_manager.cc:1398] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240501 20:01:06.394846 29851 tablet_bootstrap.cc:492] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:06.396063 29849 tablet_bootstrap.cc:654] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:06.397553 29896 raft_consensus.cc:482] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:06.398190 29896 raft_consensus.cc:504] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.399479 29557 catalog_manager.cc:5484] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 0 to 1, leader changed from <none> to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 1 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:06.401381 29896 leader_election.cc:288] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:06.402160 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e38a2e91a3d48d09a22abce723af833" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:06.404381 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e38a2e91a3d48d09a22abce723af833" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:06.406931 29772 leader_election.cc:302] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:06.408360 29896 raft_consensus.cc:2726] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:06.416759 29989 raft_consensus.cc:482] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:06.417415 29989 raft_consensus.cc:504] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.421816 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af08384057a4d85977481472f460168" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:06.422739 29659 raft_consensus.cc:2454] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 0.
I20240501 20:01:06.424286 29698 leader_election.cc:302] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:06.425860 29914 raft_consensus.cc:2781] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:06.426380 29914 raft_consensus.cc:482] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:06.427505 29914 raft_consensus.cc:3037] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:06.432622 29989 leader_election.cc:288] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:06.434948 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af08384057a4d85977481472f460168" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:06.435904 29807 raft_consensus.cc:2454] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 0.
I20240501 20:01:06.438417 29851 tablet_bootstrap.cc:654] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:06.452319 29914 raft_consensus.cc:504] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.469635 29851 tablet_bootstrap.cc:492] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:06.470296 29851 ts_tablet_manager.cc:1367] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.076s user 0.018s sys 0.001s
I20240501 20:01:06.473676 29851 raft_consensus.cc:348] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.474637 29851 raft_consensus.cc:374] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:06.474983 29851 raft_consensus.cc:729] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:06.475596 29851 consensus_queue.cc:260] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.489787 29909 consensus_queue.cc:1035] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:01:06.490914 29851 ts_tablet_manager.cc:1398] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.020s user 0.009s sys 0.000s
I20240501 20:01:06.492247 29851 tablet_bootstrap.cc:492] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:06.500661 29808 raft_consensus.cc:3037] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:06.505098 29851 tablet_bootstrap.cc:654] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:06.508293 29909 raft_consensus.cc:482] T 010ceabf3e5a4b1c9330b80dade765bb P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:06.507833 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af08384057a4d85977481472f460168" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:06.508896 29909 raft_consensus.cc:504] T 010ceabf3e5a4b1c9330b80dade765bb P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.512717 29660 raft_consensus.cc:3037] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:06.517623 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010ceabf3e5a4b1c9330b80dade765bb" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:06.518708 29909 leader_election.cc:288] T 010ceabf3e5a4b1c9330b80dade765bb P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:06.519312 29698 leader_election.cc:341] T 010ceabf3e5a4b1c9330b80dade765bb P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:06.526101 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010ceabf3e5a4b1c9330b80dade765bb" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:06.529109 29914 leader_election.cc:288] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:06.531564 29660 raft_consensus.cc:2454] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
I20240501 20:01:06.537035 29852 tablet_bootstrap.cc:492] T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:06.537515 29852 ts_tablet_manager.cc:1367] T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.177s user 0.058s sys 0.076s
I20240501 20:01:06.538012 29698 leader_election.cc:302] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:06.540015 29852 raft_consensus.cc:348] T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.540525 29849 tablet_bootstrap.cc:492] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:06.540813 29852 raft_consensus.cc:374] T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:06.541229 29849 ts_tablet_manager.cc:1367] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.159s user 0.057s sys 0.070s
I20240501 20:01:06.541152 29852 raft_consensus.cc:729] T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:06.542183 29852 consensus_queue.cc:260] T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.543258 29977 raft_consensus.cc:482] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:06.543701 29977 raft_consensus.cc:504] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.544409 29852 ts_tablet_manager.cc:1398] T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.007s user 0.005s sys 0.003s
I20240501 20:01:06.544266 29849 raft_consensus.cc:348] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.545325 29852 tablet_bootstrap.cc:492] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:06.545243 29849 raft_consensus.cc:374] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:06.545711 29977 leader_election.cc:288] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:06.545879 29849 raft_consensus.cc:729] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:06.546635 29849 consensus_queue.cc:260] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.542670 29989 raft_consensus.cc:2781] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:06.548622 29989 raft_consensus.cc:686] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:06.548918 29849 ts_tablet_manager.cc:1398] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240501 20:01:06.549521 29989 consensus_queue.cc:237] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.550172 29849 tablet_bootstrap.cc:492] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:06.551872 29852 tablet_bootstrap.cc:654] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
W20240501 20:01:06.541299 29700 leader_election.cc:341] T 010ceabf3e5a4b1c9330b80dade765bb P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:06.552845 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "456035a1cb3f4f7da52d0fc1dd1c656c" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:06.553758 29735 raft_consensus.cc:2454] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 0.
I20240501 20:01:06.557428 29624 leader_election.cc:302] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:06.557610 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "456035a1cb3f4f7da52d0fc1dd1c656c" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:06.558399 29978 raft_consensus.cc:2781] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:06.558758 29978 raft_consensus.cc:482] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:06.559067 29978 raft_consensus.cc:3037] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:06.559257 29849 tablet_bootstrap.cc:654] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:06.559556 29700 leader_election.cc:302] T 010ceabf3e5a4b1c9330b80dade765bb P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:06.560722 29851 tablet_bootstrap.cc:492] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:06.561328 29851 ts_tablet_manager.cc:1367] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.069s user 0.014s sys 0.006s
I20240501 20:01:06.562201 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af08384057a4d85977481472f460168" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:06.562806 29808 raft_consensus.cc:3037] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:06.568080 29856 raft_consensus.cc:2726] T 010ceabf3e5a4b1c9330b80dade765bb P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240501 20:01:06.568655 29625 leader_election.cc:341] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:06.569823 29808 raft_consensus.cc:2454] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
I20240501 20:01:06.571298 29851 raft_consensus.cc:348] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.576467 29851 raft_consensus.cc:374] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:06.576848 29851 raft_consensus.cc:729] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:06.577688 29851 consensus_queue.cc:260] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.567634 29978 raft_consensus.cc:504] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:06.584817 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 43560.3us
I20240501 20:01:06.586319 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "456035a1cb3f4f7da52d0fc1dd1c656c" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:06.586704 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "456035a1cb3f4f7da52d0fc1dd1c656c" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:06.587206 29735 raft_consensus.cc:3037] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
W20240501 20:01:06.588264 29625 leader_election.cc:341] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:06.584507 29978 leader_election.cc:288] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:06.595083 29851 ts_tablet_manager.cc:1398] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.033s user 0.008s sys 0.000s
I20240501 20:01:06.595984 29851 tablet_bootstrap.cc:492] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:06.596558 29896 consensus_queue.cc:1035] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:01:06.609176 29557 catalog_manager.cc:5484] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 0 to 1, leader changed from <none> to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 1 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:06.612723 29896 consensus_queue.cc:1035] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:06.616230 29849 tablet_bootstrap.cc:492] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:06.616801 29849 ts_tablet_manager.cc:1367] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.067s user 0.007s sys 0.008s
I20240501 20:01:06.625118 29849 raft_consensus.cc:348] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.625880 29849 raft_consensus.cc:374] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:06.626302 29849 raft_consensus.cc:729] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:06.627318 29852 tablet_bootstrap.cc:492] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:06.626992 29849 consensus_queue.cc:260] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.628650 29849 ts_tablet_manager.cc:1398] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20240501 20:01:06.629078 29852 ts_tablet_manager.cc:1367] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.084s user 0.009s sys 0.004s
I20240501 20:01:06.629444 29977 raft_consensus.cc:482] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:06.629987 29977 raft_consensus.cc:504] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.631719 29849 tablet_bootstrap.cc:492] T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:06.633358 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9472c53f79ec423f82c396bd980674e2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
W20240501 20:01:06.634941 29624 leader_election.cc:341] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:06.635684 29852 raft_consensus.cc:348] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.631909 29977 leader_election.cc:288] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:06.636392 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9472c53f79ec423f82c396bd980674e2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:06.642146 29852 raft_consensus.cc:374] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:06.642257 29808 raft_consensus.cc:2454] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 0.
I20240501 20:01:06.642531 29852 raft_consensus.cc:729] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:06.638561 29896 raft_consensus.cc:482] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:06.643348 29896 raft_consensus.cc:504] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.644254 29625 leader_election.cc:302] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters: fdfeac4190e045f290e003b2298d988d
I20240501 20:01:06.645291 29896 leader_election.cc:288] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:06.646246 29977 raft_consensus.cc:2781] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:06.643618 29852 consensus_queue.cc:260] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.646631 29977 raft_consensus.cc:482] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:06.646798 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcf30cf66714efdac997862dafbe4a2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:06.647457 29734 raft_consensus.cc:2454] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 0.
I20240501 20:01:06.648403 29852 ts_tablet_manager.cc:1398] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20240501 20:01:06.647017 29977 raft_consensus.cc:3037] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:06.641130 29851 tablet_bootstrap.cc:654] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:06.649716 29852 tablet_bootstrap.cc:492] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:06.651419 29773 leader_election.cc:302] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:06.641711 29735 raft_consensus.cc:2454] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:06.653156 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcf30cf66714efdac997862dafbe4a2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:06.653412 29896 raft_consensus.cc:2781] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:06.656159 29896 raft_consensus.cc:482] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240501 20:01:06.656322 29772 leader_election.cc:341] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:06.656517 29896 raft_consensus.cc:3037] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:06.665709 29896 raft_consensus.cc:504] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.666587 29977 raft_consensus.cc:504] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.668072 29624 leader_election.cc:302] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:06.669139 29979 raft_consensus.cc:2781] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:06.669603 29979 raft_consensus.cc:686] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:06.670966 29985 raft_consensus.cc:482] T 48630ae6eb394ac1b54955dd01cf0ed2 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:06.670346 29979 consensus_queue.cc:237] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.671491 29985 raft_consensus.cc:504] T 48630ae6eb394ac1b54955dd01cf0ed2 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.673619 29985 leader_election.cc:288] T 48630ae6eb394ac1b54955dd01cf0ed2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:06.672689 29896 leader_election.cc:288] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:06.675060 29849 tablet_bootstrap.cc:654] T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:06.676088 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48630ae6eb394ac1b54955dd01cf0ed2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:06.677421 29852 tablet_bootstrap.cc:654] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
W20240501 20:01:06.678401 29773 leader_election.cc:341] T 48630ae6eb394ac1b54955dd01cf0ed2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:06.680006 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcf30cf66714efdac997862dafbe4a2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
W20240501 20:01:06.682097 29772 leader_election.cc:341] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:06.683112 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48630ae6eb394ac1b54955dd01cf0ed2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:06.683897 29659 raft_consensus.cc:2454] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 0.
I20240501 20:01:06.685242 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcf30cf66714efdac997862dafbe4a2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:06.685988 29732 raft_consensus.cc:3037] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:06.688695 29772 leader_election.cc:302] T 48630ae6eb394ac1b54955dd01cf0ed2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters: fdfeac4190e045f290e003b2298d988d
I20240501 20:01:06.690955 29989 consensus_queue.cc:1035] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:06.692735 29896 raft_consensus.cc:2781] T 48630ae6eb394ac1b54955dd01cf0ed2 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:06.693123 29896 raft_consensus.cc:482] T 48630ae6eb394ac1b54955dd01cf0ed2 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:06.693425 29896 raft_consensus.cc:3037] T 48630ae6eb394ac1b54955dd01cf0ed2 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:06.697345 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9472c53f79ec423f82c396bd980674e2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
W20240501 20:01:06.701568 29624 leader_election.cc:341] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:06.706089 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9472c53f79ec423f82c396bd980674e2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:06.706692 29808 raft_consensus.cc:3037] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:06.709548 29977 leader_election.cc:288] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:06.716676 29989 consensus_queue.cc:1035] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:06.722934 29554 catalog_manager.cc:5484] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 1 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:06.732653 29852 tablet_bootstrap.cc:492] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:06.735219 29852 ts_tablet_manager.cc:1367] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.086s user 0.013s sys 0.000s
I20240501 20:01:06.737330 29732 raft_consensus.cc:2454] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
I20240501 20:01:06.739223 29849 tablet_bootstrap.cc:492] T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:06.739791 29849 ts_tablet_manager.cc:1367] T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.108s user 0.027s sys 0.001s
I20240501 20:01:06.742059 29851 tablet_bootstrap.cc:492] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:06.742493 29851 ts_tablet_manager.cc:1367] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.147s user 0.012s sys 0.021s
I20240501 20:01:06.742777 29849 raft_consensus.cc:348] T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.743489 29849 raft_consensus.cc:374] T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:06.743738 29849 raft_consensus.cc:729] T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:06.744623 29849 consensus_queue.cc:260] T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.732764 29808 raft_consensus.cc:2454] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:06.745079 29851 raft_consensus.cc:348] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.750212 29851 raft_consensus.cc:374] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:06.750510 29851 raft_consensus.cc:729] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:06.750629 29849 ts_tablet_manager.cc:1398] T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.011s user 0.006s sys 0.002s
I20240501 20:01:06.748205 29852 raft_consensus.cc:348] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.751575 29852 raft_consensus.cc:374] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:06.748703 29625 leader_election.cc:302] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters: fdfeac4190e045f290e003b2298d988d
I20240501 20:01:06.751967 29852 raft_consensus.cc:729] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:06.745196 29773 leader_election.cc:302] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:06.752744 29852 consensus_queue.cc:260] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.751180 29851 consensus_queue.cc:260] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.753190 29977 raft_consensus.cc:2781] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:06.754307 29985 raft_consensus.cc:2781] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:06.754366 29979 consensus_queue.cc:1035] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:06.754910 29977 raft_consensus.cc:686] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:06.755172 29852 ts_tablet_manager.cc:1398] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20240501 20:01:06.755780 29977 consensus_queue.cc:237] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.756554 29849 tablet_bootstrap.cc:492] T 456035a1cb3f4f7da52d0fc1dd1c656c P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:06.757052 29985 raft_consensus.cc:686] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:06.760493 29979 raft_consensus.cc:482] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:06.760968 29979 raft_consensus.cc:504] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.762341 29849 tablet_bootstrap.cc:654] T 456035a1cb3f4f7da52d0fc1dd1c656c P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:06.762126 29978 consensus_queue.cc:1035] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:06.763034 29979 leader_election.cc:288] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:06.763700 29851 ts_tablet_manager.cc:1398] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.021s user 0.007s sys 0.000s
I20240501 20:01:06.764310 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0374bb790ed345ed90452595f71383a2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:06.764657 29851 tablet_bootstrap.cc:492] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
W20240501 20:01:06.765749 29624 leader_election.cc:341] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:06.766254 29852 tablet_bootstrap.cc:492] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:06.767863 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0374bb790ed345ed90452595f71383a2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:06.768836 29625 leader_election.cc:341] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:06.769166 29625 leader_election.cc:302] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:06.769721 29977 raft_consensus.cc:2726] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:06.770601 29851 tablet_bootstrap.cc:654] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:06.772114 29852 tablet_bootstrap.cc:654] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:06.774799 29849 tablet_bootstrap.cc:492] T 456035a1cb3f4f7da52d0fc1dd1c656c P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:06.775270 29849 ts_tablet_manager.cc:1367] T 456035a1cb3f4f7da52d0fc1dd1c656c P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.004s
I20240501 20:01:06.777938 29849 raft_consensus.cc:348] T 456035a1cb3f4f7da52d0fc1dd1c656c P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.778791 29849 raft_consensus.cc:374] T 456035a1cb3f4f7da52d0fc1dd1c656c P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:06.779135 29849 raft_consensus.cc:729] T 456035a1cb3f4f7da52d0fc1dd1c656c P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:06.781855 29914 consensus_queue.cc:1035] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:06.782246 29852 tablet_bootstrap.cc:492] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:06.782151 29851 tablet_bootstrap.cc:492] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:06.782730 29852 ts_tablet_manager.cc:1367] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.004s
I20240501 20:01:06.782730 29851 ts_tablet_manager.cc:1367] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.018s user 0.017s sys 0.000s
I20240501 20:01:06.739291 29896 raft_consensus.cc:504] T 48630ae6eb394ac1b54955dd01cf0ed2 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.786008 29852 raft_consensus.cc:348] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.786893 29852 raft_consensus.cc:374] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:06.787220 29852 raft_consensus.cc:729] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:06.787242 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48630ae6eb394ac1b54955dd01cf0ed2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:06.787992 29660 raft_consensus.cc:3037] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:06.788121 29852 consensus_queue.cc:260] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.779969 29849 consensus_queue.cc:260] T 456035a1cb3f4f7da52d0fc1dd1c656c P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.791263 29849 ts_tablet_manager.cc:1398] T 456035a1cb3f4f7da52d0fc1dd1c656c P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.016s user 0.002s sys 0.002s
I20240501 20:01:06.792201 29849 tablet_bootstrap.cc:492] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:06.792867 29852 ts_tablet_manager.cc:1398] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.010s user 0.006s sys 0.002s
I20240501 20:01:06.793699 29852 tablet_bootstrap.cc:492] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:06.794543 29660 raft_consensus.cc:2454] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
I20240501 20:01:06.757884 29985 consensus_queue.cc:237] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.795732 29772 leader_election.cc:302] T 48630ae6eb394ac1b54955dd01cf0ed2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:06.796604 29984 raft_consensus.cc:2781] T 48630ae6eb394ac1b54955dd01cf0ed2 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:06.797283 29984 raft_consensus.cc:686] T 48630ae6eb394ac1b54955dd01cf0ed2 P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:06.798214 29977 raft_consensus.cc:482] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:06.798327 29984 consensus_queue.cc:237] T 48630ae6eb394ac1b54955dd01cf0ed2 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.798682 29977 raft_consensus.cc:504] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.799219 29896 leader_election.cc:288] T 48630ae6eb394ac1b54955dd01cf0ed2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:06.799737 29852 tablet_bootstrap.cc:654] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:06.803400 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48630ae6eb394ac1b54955dd01cf0ed2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:06.804627 29732 raft_consensus.cc:3037] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:06.801672 29851 raft_consensus.cc:348] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.805332 29851 raft_consensus.cc:374] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:06.805908 29851 raft_consensus.cc:729] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:06.806960 29851 consensus_queue.cc:260] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.813268 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1a08b455b94ae089396a15c837ddc2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
W20240501 20:01:06.815522 29624 leader_election.cc:341] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:06.816141 29849 tablet_bootstrap.cc:654] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:06.816432 29977 leader_election.cc:288] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:06.824090 29851 ts_tablet_manager.cc:1398] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.041s user 0.006s sys 0.000s
I20240501 20:01:06.829198 29851 tablet_bootstrap.cc:492] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:06.840023 29557 catalog_manager.cc:5484] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 0 to 1, leader changed from <none> to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 1 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:06.840610 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1a08b455b94ae089396a15c837ddc2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:06.841576 29732 raft_consensus.cc:2454] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
I20240501 20:01:06.848114 29849 tablet_bootstrap.cc:492] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:06.848584 29849 ts_tablet_manager.cc:1367] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.057s user 0.016s sys 0.007s
I20240501 20:01:06.851202 29989 raft_consensus.cc:482] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:06.851729 29989 raft_consensus.cc:504] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.855624 29989 leader_election.cc:288] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:06.859884 29851 tablet_bootstrap.cc:654] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:06.864506 29849 raft_consensus.cc:348] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.865307 29849 raft_consensus.cc:374] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:06.865636 29849 raft_consensus.cc:729] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:06.866389 29852 tablet_bootstrap.cc:492] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:06.866820 29852 ts_tablet_manager.cc:1367] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.073s user 0.016s sys 0.008s
I20240501 20:01:06.866210 29849 consensus_queue.cc:260] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.868726 29849 ts_tablet_manager.cc:1398] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
W20240501 20:01:06.858251 29625 leader_election.cc:341] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:06.869446 29625 leader_election.cc:302] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:06.870697 29979 raft_consensus.cc:2726] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:06.872004 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9472c53f79ec423f82c396bd980674e2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:06.873196 29849 tablet_bootstrap.cc:492] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:06.874087 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9472c53f79ec423f82c396bd980674e2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:06.874886 29806 raft_consensus.cc:2379] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d in current term 1: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:06.874469 29987 raft_consensus.cc:482] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:06.876025 29554 catalog_manager.cc:5484] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 1 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:06.876156 29987 raft_consensus.cc:504] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.882354 29660 raft_consensus.cc:3037] T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:06.883235 29849 tablet_bootstrap.cc:654] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:06.884009 29987 leader_election.cc:288] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:06.878448 29700 leader_election.cc:302] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:06.886312 29909 raft_consensus.cc:3037] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:06.882555 29556 catalog_manager.cc:5484] T 48630ae6eb394ac1b54955dd01cf0ed2 P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 0 to 1, leader changed from <none> to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 1 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:06.887847 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458740ab5c444b6ab3d1991c69db5084" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
W20240501 20:01:06.889142 29698 leader_election.cc:341] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:06.894407 29988 consensus_queue.cc:1035] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:01:06.896461 29977 raft_consensus.cc:482] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:06.896961 29977 raft_consensus.cc:504] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.879861 29852 raft_consensus.cc:348] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.898790 29852 raft_consensus.cc:374] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:06.898862 29977 leader_election.cc:288] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:06.899135 29852 raft_consensus.cc:729] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:06.899758 29852 consensus_queue.cc:260] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.900425 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c597f15e844f30837012baed4b0501" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:06.901079 29735 raft_consensus.cc:2454] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 0.
I20240501 20:01:06.901683 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c597f15e844f30837012baed4b0501" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:06.902305 29807 raft_consensus.cc:2454] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 0.
I20240501 20:01:06.902236 29624 leader_election.cc:302] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:06.886301 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458740ab5c444b6ab3d1991c69db5084" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:06.905786 29977 raft_consensus.cc:2781] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:06.906105 29977 raft_consensus.cc:482] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:06.903640 29806 raft_consensus.cc:2454] T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 0.
I20240501 20:01:06.906361 29977 raft_consensus.cc:3037] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:06.910185 29700 leader_election.cc:302] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:06.911407 29852 ts_tablet_manager.cc:1398] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.044s user 0.007s sys 0.000s
I20240501 20:01:06.912111 29856 raft_consensus.cc:2781] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:06.912328 29852 tablet_bootstrap.cc:492] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:06.912662 29856 raft_consensus.cc:482] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:06.913177 29856 raft_consensus.cc:3037] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:06.920737 29851 tablet_bootstrap.cc:492] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:06.921212 29851 ts_tablet_manager.cc:1367] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.092s user 0.009s sys 0.000s
I20240501 20:01:06.924517 29851 raft_consensus.cc:348] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.925211 29851 raft_consensus.cc:374] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:06.925473 29851 raft_consensus.cc:729] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:06.926054 29851 consensus_queue.cc:260] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.927537 29851 ts_tablet_manager.cc:1398] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240501 20:01:06.928449 29851 tablet_bootstrap.cc:492] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:06.939546 29852 tablet_bootstrap.cc:654] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:06.940227 29909 raft_consensus.cc:2726] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:06.958015 29849 tablet_bootstrap.cc:492] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:06.958546 29849 ts_tablet_manager.cc:1367] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.086s user 0.028s sys 0.029s
I20240501 20:01:06.957985 29856 raft_consensus.cc:504] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.964166 29856 leader_election.cc:288] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:06.964781 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458740ab5c444b6ab3d1991c69db5084" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
W20240501 20:01:06.967085 29698 leader_election.cc:341] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:06.969938 29977 raft_consensus.cc:504] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.972848 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c597f15e844f30837012baed4b0501" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:06.973415 29806 raft_consensus.cc:3037] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:06.983382 29851 tablet_bootstrap.cc:654] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:06.986871 29849 raft_consensus.cc:348] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.987632 29849 raft_consensus.cc:374] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:06.987684 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c597f15e844f30837012baed4b0501" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:06.988394 29977 leader_election.cc:288] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:06.988385 29735 raft_consensus.cc:3037] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:06.989648 29806 raft_consensus.cc:2454] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:06.987933 29849 raft_consensus.cc:729] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:06.990767 29849 consensus_queue.cc:260] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:06.993335 29849 ts_tablet_manager.cc:1398] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.034s user 0.007s sys 0.000s
I20240501 20:01:06.994575 29849 tablet_bootstrap.cc:492] T 010ceabf3e5a4b1c9330b80dade765bb P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:07.001761 29735 raft_consensus.cc:2454] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:07.003921 29849 tablet_bootstrap.cc:654] T 010ceabf3e5a4b1c9330b80dade765bb P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:07.005038 29625 leader_election.cc:302] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:07.007763 29977 raft_consensus.cc:482] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:07.014796 29977 raft_consensus.cc:504] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.017405 29977 leader_election.cc:288] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:07.012595 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458740ab5c444b6ab3d1991c69db5084" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:07.018201 29806 raft_consensus.cc:3037] T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:07.019480 29979 raft_consensus.cc:2781] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:07.019951 29979 raft_consensus.cc:686] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:07.021173 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcf30cf66714efdac997862dafbe4a2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:07.020885 29979 consensus_queue.cc:237] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.032895 29849 tablet_bootstrap.cc:492] T 010ceabf3e5a4b1c9330b80dade765bb P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:07.027742 29806 raft_consensus.cc:2454] T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
I20240501 20:01:07.033390 29849 ts_tablet_manager.cc:1367] T 010ceabf3e5a4b1c9330b80dade765bb P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.039s user 0.015s sys 0.000s
I20240501 20:01:07.033864 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcf30cf66714efdac997862dafbe4a2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:07.034493 29700 leader_election.cc:302] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:07.035576 29856 raft_consensus.cc:2781] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:07.033790 29852 tablet_bootstrap.cc:492] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:07.036216 29856 raft_consensus.cc:686] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:07.036353 29852 ts_tablet_manager.cc:1367] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.124s user 0.008s sys 0.027s
I20240501 20:01:07.036731 29851 tablet_bootstrap.cc:492] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:07.037142 29851 ts_tablet_manager.cc:1367] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.109s user 0.018s sys 0.027s
I20240501 20:01:07.037036 29856 consensus_queue.cc:237] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.039757 29985 raft_consensus.cc:482] T 456035a1cb3f4f7da52d0fc1dd1c656c P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:07.039829 29852 raft_consensus.cc:348] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.040647 29852 raft_consensus.cc:374] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:07.040329 29985 raft_consensus.cc:504] T 456035a1cb3f4f7da52d0fc1dd1c656c P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.040910 29852 raft_consensus.cc:729] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:07.041527 29852 consensus_queue.cc:260] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.042382 29985 leader_election.cc:288] T 456035a1cb3f4f7da52d0fc1dd1c656c P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:07.038625 29849 raft_consensus.cc:348] T 010ceabf3e5a4b1c9330b80dade765bb P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.044586 29849 raft_consensus.cc:374] T 010ceabf3e5a4b1c9330b80dade765bb P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:07.044719 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "456035a1cb3f4f7da52d0fc1dd1c656c" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:07.045500 29852 ts_tablet_manager.cc:1398] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20240501 20:01:07.045832 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "456035a1cb3f4f7da52d0fc1dd1c656c" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:07.045143 29851 raft_consensus.cc:348] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.046384 29852 tablet_bootstrap.cc:492] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:07.046587 29732 raft_consensus.cc:2379] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 1: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:07.044874 29849 raft_consensus.cc:729] T 010ceabf3e5a4b1c9330b80dade765bb P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:07.048182 29773 leader_election.cc:302] T 456035a1cb3f4f7da52d0fc1dd1c656c P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:07.048947 29849 consensus_queue.cc:260] T 010ceabf3e5a4b1c9330b80dade765bb P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.049934 29985 raft_consensus.cc:3037] T 456035a1cb3f4f7da52d0fc1dd1c656c P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:07.052822 29852 tablet_bootstrap.cc:654] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:07.055022 29849 ts_tablet_manager.cc:1398] T 010ceabf3e5a4b1c9330b80dade765bb P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.021s user 0.005s sys 0.000s
I20240501 20:01:07.056161 29849 tablet_bootstrap.cc:492] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:07.058068 29735 raft_consensus.cc:2379] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 1: Already voted for candidate ee0d97247d394a8facfa774f15f3a67d in this term.
I20240501 20:01:07.059525 29624 leader_election.cc:302] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:07.061311 29978 raft_consensus.cc:3037] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:07.063401 29909 consensus_queue.cc:1035] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:01:07.064500 29852 tablet_bootstrap.cc:492] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:07.064958 29852 ts_tablet_manager.cc:1367] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.004s
I20240501 20:01:07.067360 29852 raft_consensus.cc:348] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.068030 29852 raft_consensus.cc:374] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:07.068318 29852 raft_consensus.cc:729] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:07.068892 29978 raft_consensus.cc:2726] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:07.069096 29852 consensus_queue.cc:260] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.074189 29909 consensus_queue.cc:1035] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:01:07.074347 29852 ts_tablet_manager.cc:1398] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20240501 20:01:07.074803 29896 raft_consensus.cc:482] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:07.077119 29896 raft_consensus.cc:504] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.046556 29851 raft_consensus.cc:374] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:07.074915 29557 catalog_manager.cc:5484] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 1 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:07.078342 29851 raft_consensus.cc:729] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:07.079304 29851 consensus_queue.cc:260] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.083484 29896 leader_election.cc:288] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:07.085009 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb27193abbf4644a48662909bc7b04a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
W20240501 20:01:07.086558 29772 leader_election.cc:341] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:07.089766 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb27193abbf4644a48662909bc7b04a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
W20240501 20:01:07.091066 29773 leader_election.cc:341] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:07.091477 29773 leader_election.cc:302] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:07.092926 29896 raft_consensus.cc:2726] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:07.093796 29849 tablet_bootstrap.cc:654] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:07.095363 29851 ts_tablet_manager.cc:1398] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.058s user 0.008s sys 0.012s
I20240501 20:01:07.096247 29851 tablet_bootstrap.cc:492] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:07.098031 29985 raft_consensus.cc:2726] T 456035a1cb3f4f7da52d0fc1dd1c656c P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:07.099126 29977 raft_consensus.cc:482] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:07.099582 29977 raft_consensus.cc:504] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.100557 29987 raft_consensus.cc:482] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:07.100976 29987 raft_consensus.cc:504] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.101445 29851 tablet_bootstrap.cc:654] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:07.102545 29977 leader_election.cc:288] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:07.103063 29987 leader_election.cc:288] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:07.104629 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a2bba62f434d1c8489b55c8fec2187" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:07.105926 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a2bba62f434d1c8489b55c8fec2187" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
W20240501 20:01:07.108172 29624 leader_election.cc:341] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Illegal state: must be running to vote when last-logged opid is not known
W20240501 20:01:07.109863 29625 leader_election.cc:341] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:07.110419 29625 leader_election.cc:302] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:07.112089 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07bdb8b09b8c4ef4b8e49daa9b8c01d6" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:07.113762 29700 leader_election.cc:341] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:07.077426 29852 tablet_bootstrap.cc:492] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:07.116258 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07bdb8b09b8c4ef4b8e49daa9b8c01d6" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:07.116910 29660 raft_consensus.cc:2454] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 0.
I20240501 20:01:07.118026 29698 leader_election.cc:302] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:07.116253 29978 raft_consensus.cc:2726] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:07.118860 29987 raft_consensus.cc:2781] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:07.119277 29987 raft_consensus.cc:482] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:07.119638 29987 raft_consensus.cc:3037] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:07.123313 29852 tablet_bootstrap.cc:654] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:07.127079 29557 catalog_manager.cc:5484] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 0 to 1, leader changed from <none> to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 1 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:07.130745 29851 tablet_bootstrap.cc:492] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:07.131194 29851 ts_tablet_manager.cc:1367] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.035s user 0.018s sys 0.015s
I20240501 20:01:07.134234 29851 raft_consensus.cc:348] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.134922 29851 raft_consensus.cc:374] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:07.135183 29851 raft_consensus.cc:729] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:07.135752 29851 consensus_queue.cc:260] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.137725 29849 tablet_bootstrap.cc:492] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:07.138182 29852 tablet_bootstrap.cc:492] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:07.138233 29849 ts_tablet_manager.cc:1367] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.082s user 0.012s sys 0.004s
I20240501 20:01:07.138746 29852 ts_tablet_manager.cc:1367] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.062s user 0.008s sys 0.000s
I20240501 20:01:07.141034 29849 raft_consensus.cc:348] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.141952 29849 raft_consensus.cc:374] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:07.142274 29849 raft_consensus.cc:729] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:07.142100 29852 raft_consensus.cc:348] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.142830 29852 raft_consensus.cc:374] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:07.143105 29852 raft_consensus.cc:729] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:07.143286 29849 consensus_queue.cc:260] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.145860 29851 ts_tablet_manager.cc:1398] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.014s user 0.004s sys 0.004s
I20240501 20:01:07.146198 29849 ts_tablet_manager.cc:1398] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20240501 20:01:07.146710 29851 tablet_bootstrap.cc:492] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:07.147675 29849 tablet_bootstrap.cc:492] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:07.143699 29852 consensus_queue.cc:260] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.154990 29852 ts_tablet_manager.cc:1398] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.016s user 0.006s sys 0.001s
I20240501 20:01:07.157076 29851 tablet_bootstrap.cc:654] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:07.159816 29852 tablet_bootstrap.cc:492] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:07.154871 29987 raft_consensus.cc:504] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.168184 29909 raft_consensus.cc:482] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:07.171866 29978 consensus_queue.cc:1035] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:07.172820 29909 raft_consensus.cc:504] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.179761 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77be7101c0fe4085b6706296f609870a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
W20240501 20:01:07.181115 29698 leader_election.cc:341] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:07.181479 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77be7101c0fe4085b6706296f609870a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:07.182965 29700 leader_election.cc:341] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:07.184005 29700 leader_election.cc:302] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:07.203941 29988 raft_consensus.cc:2726] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:07.191462 29978 consensus_queue.cc:1035] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.024s
I20240501 20:01:07.178369 29909 leader_election.cc:288] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:07.218919 29896 raft_consensus.cc:482] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:07.219437 29896 raft_consensus.cc:504] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.221366 29896 leader_election.cc:288] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:07.222404 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b337b8e3e38d4ad6b1076fbd0d0a1609" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
W20240501 20:01:07.228442 29773 leader_election.cc:341] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:07.229203 29851 tablet_bootstrap.cc:492] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:07.229667 29851 ts_tablet_manager.cc:1367] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.083s user 0.021s sys 0.045s
I20240501 20:01:07.233547 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b337b8e3e38d4ad6b1076fbd0d0a1609" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:07.233808 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07bdb8b09b8c4ef4b8e49daa9b8c01d6" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:07.234336 29656 raft_consensus.cc:3037] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:07.280242 29978 consensus_queue.cc:1035] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20240501 20:01:07.281222 29772 leader_election.cc:341] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:07.253883 29987 leader_election.cc:288] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:07.316706 29849 tablet_bootstrap.cc:654] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:07.275883 29656 raft_consensus.cc:2454] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
I20240501 20:01:07.400136 29851 raft_consensus.cc:348] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.401335 29851 raft_consensus.cc:374] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:07.401664 29851 raft_consensus.cc:729] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:07.402343 29851 consensus_queue.cc:260] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:07.330403 29606 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240501 20:01:07.408129 29852 tablet_bootstrap.cc:654] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:07.330987 29979 raft_consensus.cc:482] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:07.408954 29985 consensus_queue.cc:1035] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240501 20:01:07.411929 29851 ts_tablet_manager.cc:1398] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.182s user 0.020s sys 0.064s
I20240501 20:01:07.412534 29698 leader_election.cc:302] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:07.415387 29851 tablet_bootstrap.cc:492] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:07.409478 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07bdb8b09b8c4ef4b8e49daa9b8c01d6" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:07.313745 29772 leader_election.cc:302] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:07.419533 29987 raft_consensus.cc:2781] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:07.418159 29978 consensus_queue.cc:1035] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20240501 20:01:07.422896 29987 raft_consensus.cc:686] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:07.424597 29985 raft_consensus.cc:2726] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:07.415149 29979 raft_consensus.cc:504] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.420490 29808 raft_consensus.cc:3037] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
W20240501 20:01:07.425941 29772 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 144753us
I20240501 20:01:07.423712 29987 consensus_queue.cc:237] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.465413 29852 tablet_bootstrap.cc:492] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:07.465886 29852 ts_tablet_manager.cc:1367] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.306s user 0.020s sys 0.013s
I20240501 20:01:07.468600 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb27193abbf4644a48662909bc7b04a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:07.469250 29732 raft_consensus.cc:2454] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 0.
I20240501 20:01:07.470147 29979 leader_election.cc:288] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:07.470572 29624 leader_election.cc:302] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:07.473440 29852 raft_consensus.cc:348] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.474051 29909 raft_consensus.cc:482] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:07.474625 29909 raft_consensus.cc:504] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.474143 29852 raft_consensus.cc:374] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:07.476511 29852 raft_consensus.cc:729] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:07.477186 29852 consensus_queue.cc:260] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.478412 29985 raft_consensus.cc:482] T 010ceabf3e5a4b1c9330b80dade765bb P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:07.478757 29984 consensus_queue.cc:1035] T 48630ae6eb394ac1b54955dd01cf0ed2 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:07.478993 29852 ts_tablet_manager.cc:1398] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.013s user 0.004s sys 0.000s
I20240501 20:01:07.479880 29852 tablet_bootstrap.cc:492] T 010ceabf3e5a4b1c9330b80dade765bb P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:07.478931 29909 leader_election.cc:288] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:07.479470 30065 raft_consensus.cc:2781] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:07.480859 30065 raft_consensus.cc:482] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:07.481184 30065 raft_consensus.cc:3037] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:07.478883 29985 raft_consensus.cc:504] T 010ceabf3e5a4b1c9330b80dade765bb P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.487928 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b337b8e3e38d4ad6b1076fbd0d0a1609" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
W20240501 20:01:07.489576 29698 leader_election.cc:341] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:07.497321 30065 raft_consensus.cc:504] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.499234 30065 leader_election.cc:288] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:07.500412 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb27193abbf4644a48662909bc7b04a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:07.500711 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b337b8e3e38d4ad6b1076fbd0d0a1609" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:07.501062 29733 raft_consensus.cc:3037] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:07.501390 29807 raft_consensus.cc:2454] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 0.
I20240501 20:01:07.507122 29849 tablet_bootstrap.cc:492] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:07.509363 29849 ts_tablet_manager.cc:1367] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.362s user 0.020s sys 0.013s
I20240501 20:01:07.509936 29852 tablet_bootstrap.cc:654] T 010ceabf3e5a4b1c9330b80dade765bb P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:07.516335 29808 raft_consensus.cc:2454] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
I20240501 20:01:07.520891 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb27193abbf4644a48662909bc7b04a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:07.521868 29804 raft_consensus.cc:2454] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 0.
I20240501 20:01:07.512465 29849 raft_consensus.cc:348] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.522838 29849 raft_consensus.cc:374] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:07.523058 29849 raft_consensus.cc:729] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:07.523586 29849 consensus_queue.cc:260] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.526593 29851 tablet_bootstrap.cc:654] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:07.529388 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010ceabf3e5a4b1c9330b80dade765bb" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:07.530267 29734 raft_consensus.cc:2454] T 010ceabf3e5a4b1c9330b80dade765bb P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 0.
I20240501 20:01:07.524924 29985 leader_election.cc:288] T 010ceabf3e5a4b1c9330b80dade765bb P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:07.545356 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010ceabf3e5a4b1c9330b80dade765bb" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:07.537707 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb27193abbf4644a48662909bc7b04a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:07.560346 29852 tablet_bootstrap.cc:492] T 010ceabf3e5a4b1c9330b80dade765bb P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:07.560853 29852 ts_tablet_manager.cc:1367] T 010ceabf3e5a4b1c9330b80dade765bb P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.081s user 0.011s sys 0.000s
I20240501 20:01:07.562813 29849 ts_tablet_manager.cc:1398] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.053s user 0.006s sys 0.000s
I20240501 20:01:07.563961 29554 catalog_manager.cc:5484] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 0 to 1, leader changed from <none> to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 1 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:07.565907 29849 tablet_bootstrap.cc:492] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:07.564154 29804 raft_consensus.cc:3037] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:07.561745 29700 leader_election.cc:302] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:07.565382 29984 consensus_queue.cc:1035] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20240501 20:01:07.572013 29773 leader_election.cc:302] T 010ceabf3e5a4b1c9330b80dade765bb P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:07.571897 29896 consensus_queue.cc:1035] T 48630ae6eb394ac1b54955dd01cf0ed2 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20240501 20:01:07.572685 29914 raft_consensus.cc:2781] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:07.580722 29914 raft_consensus.cc:482] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:07.581089 29914 raft_consensus.cc:3037] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:07.581847 29984 raft_consensus.cc:2781] T 010ceabf3e5a4b1c9330b80dade765bb P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:07.582384 29984 raft_consensus.cc:482] T 010ceabf3e5a4b1c9330b80dade765bb P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:07.582815 29984 raft_consensus.cc:3037] T 010ceabf3e5a4b1c9330b80dade765bb P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:07.577788 29852 raft_consensus.cc:348] T 010ceabf3e5a4b1c9330b80dade765bb P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.588897 29852 raft_consensus.cc:374] T 010ceabf3e5a4b1c9330b80dade765bb P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:07.589241 29852 raft_consensus.cc:729] T 010ceabf3e5a4b1c9330b80dade765bb P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:07.589818 29852 consensus_queue.cc:260] T 010ceabf3e5a4b1c9330b80dade765bb P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.591554 29852 ts_tablet_manager.cc:1398] T 010ceabf3e5a4b1c9330b80dade765bb P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.030s user 0.004s sys 0.000s
I20240501 20:01:07.592291 29852 tablet_bootstrap.cc:492] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:07.588431 30063 consensus_queue.cc:1035] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20240501 20:01:07.600950 29852 tablet_bootstrap.cc:654] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
W20240501 20:01:07.602670 29772 leader_election.cc:341] T 010ceabf3e5a4b1c9330b80dade765bb P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:07.608285 29909 consensus_queue.cc:1035] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:07.618423 29733 raft_consensus.cc:2454] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
W20240501 20:01:07.619887 29698 consensus_peers.cc:485] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d -> Peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Couldn't send request to peer 4c8da1d580614449a636b855bf11b95f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:07.619980 29624 leader_election.cc:302] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:07.621032 30063 raft_consensus.cc:2781] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:07.621498 30063 raft_consensus.cc:686] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:07.622332 30063 consensus_queue.cc:237] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.645246 29851 tablet_bootstrap.cc:492] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:07.645918 29851 ts_tablet_manager.cc:1367] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.231s user 0.030s sys 0.006s
I20240501 20:01:07.648839 29851 raft_consensus.cc:348] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.649566 29851 raft_consensus.cc:374] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:07.649852 29851 raft_consensus.cc:729] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:07.650104 29984 raft_consensus.cc:504] T 010ceabf3e5a4b1c9330b80dade765bb P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.650504 29851 consensus_queue.cc:260] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.651989 29984 leader_election.cc:288] T 010ceabf3e5a4b1c9330b80dade765bb P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:07.641953 29804 raft_consensus.cc:2454] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:07.652673 29851 ts_tablet_manager.cc:1398] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20240501 20:01:07.653556 29851 tablet_bootstrap.cc:492] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:07.649192 29914 raft_consensus.cc:504] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.665165 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b337b8e3e38d4ad6b1076fbd0d0a1609" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:07.673482 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b337b8e3e38d4ad6b1076fbd0d0a1609" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:07.674145 29804 raft_consensus.cc:3037] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:07.678521 29557 catalog_manager.cc:5484] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 1 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:07.669770 29914 leader_election.cc:288] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:07.682004 29698 leader_election.cc:341] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:07.684019 29856 raft_consensus.cc:482] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:07.684617 29856 raft_consensus.cc:504] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.686462 29856 leader_election.cc:288] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:07.687602 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd5e83faaff4e2da4e4cc22ba410d33" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:07.688396 29659 raft_consensus.cc:2379] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d in current term 2: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:07.689147 29804 raft_consensus.cc:2454] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
I20240501 20:01:07.694721 29700 leader_election.cc:302] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:07.698369 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd5e83faaff4e2da4e4cc22ba410d33" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:07.699051 29805 raft_consensus.cc:2454] T 8bd5e83faaff4e2da4e4cc22ba410d33 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 2.
I20240501 20:01:07.702924 29856 raft_consensus.cc:2781] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:07.703433 29856 raft_consensus.cc:686] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:07.705094 29856 consensus_queue.cc:237] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.716233 29852 tablet_bootstrap.cc:492] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:07.704540 29700 leader_election.cc:302] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:07.716334 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010ceabf3e5a4b1c9330b80dade765bb" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:07.721658 29656 raft_consensus.cc:3037] T 010ceabf3e5a4b1c9330b80dade765bb P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:07.720454 29852 ts_tablet_manager.cc:1367] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.128s user 0.018s sys 0.003s
I20240501 20:01:07.717286 30065 consensus_queue.cc:1035] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:07.729671 29984 raft_consensus.cc:482] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:07.730235 29984 raft_consensus.cc:504] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.731022 29987 raft_consensus.cc:2781] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:07.731395 29987 raft_consensus.cc:482] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:07.731765 29987 raft_consensus.cc:3037] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:07.732259 29984 leader_election.cc:288] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:07.737578 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77be7101c0fe4085b6706296f609870a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:07.737814 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010ceabf3e5a4b1c9330b80dade765bb" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:07.738327 29734 raft_consensus.cc:3037] T 010ceabf3e5a4b1c9330b80dade765bb P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:07.746176 29849 tablet_bootstrap.cc:654] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:07.748548 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77be7101c0fe4085b6706296f609870a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:07.755671 29733 raft_consensus.cc:2454] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 0.
I20240501 20:01:07.749508 29987 raft_consensus.cc:504] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.727444 29852 raft_consensus.cc:348] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.758186 29987 leader_election.cc:288] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:07.758287 29852 raft_consensus.cc:374] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:07.758651 29852 raft_consensus.cc:729] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:07.749284 29557 catalog_manager.cc:5484] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 0 to 1, leader changed from <none> to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 1 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:07.758877 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd5e83faaff4e2da4e4cc22ba410d33" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:07.759752 29659 raft_consensus.cc:2379] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d in current term 2: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:07.760473 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd5e83faaff4e2da4e4cc22ba410d33" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:07.764788 29656 raft_consensus.cc:2454] T 010ceabf3e5a4b1c9330b80dade765bb P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
I20240501 20:01:07.719031 29851 tablet_bootstrap.cc:654] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
W20240501 20:01:07.752665 29772 leader_election.cc:341] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:07.767199 29773 leader_election.cc:302] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:07.752562 29734 raft_consensus.cc:2454] T 010ceabf3e5a4b1c9330b80dade765bb P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
I20240501 20:01:07.768373 30073 raft_consensus.cc:2781] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:07.768759 30073 raft_consensus.cc:482] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:07.769143 30073 raft_consensus.cc:3037] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:07.777541 29804 raft_consensus.cc:2454] T 8bd5e83faaff4e2da4e4cc22ba410d33 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 2.
I20240501 20:01:07.779606 29700 leader_election.cc:302] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:07.781082 29987 raft_consensus.cc:2781] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:01:07.781561 29987 raft_consensus.cc:686] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 2 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:07.782434 29987 consensus_queue.cc:237] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.785826 30073 raft_consensus.cc:504] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.788605 30073 leader_election.cc:288] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:07.759233 29852 consensus_queue.cc:260] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.781311 29773 leader_election.cc:302] T 010ceabf3e5a4b1c9330b80dade765bb P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:07.793283 29896 raft_consensus.cc:2781] T 010ceabf3e5a4b1c9330b80dade765bb P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:07.798750 29896 raft_consensus.cc:686] T 010ceabf3e5a4b1c9330b80dade765bb P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:07.799613 29896 consensus_queue.cc:237] T 010ceabf3e5a4b1c9330b80dade765bb P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.802515 29852 ts_tablet_manager.cc:1398] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.080s user 0.004s sys 0.001s
I20240501 20:01:07.803445 29852 tablet_bootstrap.cc:492] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:07.805500 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77be7101c0fe4085b6706296f609870a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:07.806197 29734 raft_consensus.cc:3037] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:07.807195 29849 tablet_bootstrap.cc:492] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:07.807641 29849 ts_tablet_manager.cc:1367] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.242s user 0.013s sys 0.000s
I20240501 20:01:07.816380 29849 raft_consensus.cc:348] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.817128 29849 raft_consensus.cc:374] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:07.816808 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77be7101c0fe4085b6706296f609870a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:07.820065 29734 raft_consensus.cc:2454] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
I20240501 20:01:07.821205 29773 leader_election.cc:302] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
W20240501 20:01:07.821866 29772 leader_election.cc:341] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:07.822484 29849 raft_consensus.cc:729] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:07.822713 30073 raft_consensus.cc:2781] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:07.823295 30073 raft_consensus.cc:686] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:07.823071 29849 consensus_queue.cc:260] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.824868 29849 ts_tablet_manager.cc:1398] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.017s user 0.008s sys 0.000s
I20240501 20:01:07.824134 30073 consensus_queue.cc:237] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.825685 29849 tablet_bootstrap.cc:492] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:07.826604 29557 catalog_manager.cc:5484] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 1 to 2, leader changed from ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 2 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:07.830827 29849 tablet_bootstrap.cc:654] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:07.833611 29852 tablet_bootstrap.cc:654] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:07.832973 29851 tablet_bootstrap.cc:492] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:07.834712 29851 ts_tablet_manager.cc:1367] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.181s user 0.035s sys 0.001s
I20240501 20:01:07.837244 29851 raft_consensus.cc:348] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.838037 29851 raft_consensus.cc:374] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:07.838315 29851 raft_consensus.cc:729] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:07.838958 29851 consensus_queue.cc:260] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.845167 29851 ts_tablet_manager.cc:1398] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20240501 20:01:07.846000 29851 tablet_bootstrap.cc:492] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:07.848393 29558 catalog_manager.cc:5484] T 010ceabf3e5a4b1c9330b80dade765bb P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 0 to 1, leader changed from <none> to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 1 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:07.852133 29851 tablet_bootstrap.cc:654] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:07.859634 29849 tablet_bootstrap.cc:492] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:07.860023 29849 ts_tablet_manager.cc:1367] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.035s user 0.007s sys 0.009s
I20240501 20:01:07.862594 29849 raft_consensus.cc:348] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.863334 29849 raft_consensus.cc:374] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:07.863512 29852 tablet_bootstrap.cc:492] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:07.864393 29851 tablet_bootstrap.cc:492] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:07.864531 29852 ts_tablet_manager.cc:1367] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.061s user 0.017s sys 0.001s
I20240501 20:01:07.864864 29851 ts_tablet_manager.cc:1367] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.000s
I20240501 20:01:07.867228 29852 raft_consensus.cc:348] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.868017 29852 raft_consensus.cc:374] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:07.867686 29851 raft_consensus.cc:348] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.868499 29851 raft_consensus.cc:374] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:07.868471 29852 raft_consensus.cc:729] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:07.869002 29851 raft_consensus.cc:729] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:07.863605 29849 raft_consensus.cc:729] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:07.869485 29852 consensus_queue.cc:260] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.870227 29849 consensus_queue.cc:260] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.870121 29558 catalog_manager.cc:5484] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 0 to 1, leader changed from <none> to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 1 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:07.871639 29851 consensus_queue.cc:260] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.872279 29849 ts_tablet_manager.cc:1398] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.012s user 0.002s sys 0.004s
I20240501 20:01:07.873131 29849 tablet_bootstrap.cc:492] T 25e07220db7444ae97f294d70eda6a93 P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:07.873729 29851 ts_tablet_manager.cc:1398] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20240501 20:01:07.878324 29849 tablet_bootstrap.cc:654] T 25e07220db7444ae97f294d70eda6a93 P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:07.878866 29851 tablet_bootstrap.cc:492] T c651044a305c4168805825a53d910899 P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:07.882305 29852 ts_tablet_manager.cc:1398] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.017s user 0.001s sys 0.004s
I20240501 20:01:07.884690 29851 tablet_bootstrap.cc:654] T c651044a305c4168805825a53d910899 P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:07.887091 30063 raft_consensus.cc:482] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:07.887562 30063 raft_consensus.cc:504] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.889482 30063 leader_election.cc:288] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:07.891499 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e07220db7444ae97f294d70eda6a93" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:07.893077 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e07220db7444ae97f294d70eda6a93" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:07.887928 29987 raft_consensus.cc:482] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:07.896410 29987 raft_consensus.cc:504] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.896900 29851 tablet_bootstrap.cc:492] T c651044a305c4168805825a53d910899 P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:07.897377 29851 ts_tablet_manager.cc:1367] T c651044a305c4168805825a53d910899 P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.003s
I20240501 20:01:07.897441 29734 raft_consensus.cc:2454] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 0.
I20240501 20:01:07.897953 29852 tablet_bootstrap.cc:492] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:07.899919 29624 leader_election.cc:302] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:07.900807 30063 raft_consensus.cc:2781] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:07.901170 30063 raft_consensus.cc:482] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:07.901027 29851 raft_consensus.cc:348] T c651044a305c4168805825a53d910899 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.901540 30063 raft_consensus.cc:3037] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:07.901762 29851 raft_consensus.cc:374] T c651044a305c4168805825a53d910899 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:07.902060 29851 raft_consensus.cc:729] T c651044a305c4168805825a53d910899 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:07.902673 29851 consensus_queue.cc:260] T c651044a305c4168805825a53d910899 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:07.904695 29625 leader_election.cc:341] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:07.904783 29851 ts_tablet_manager.cc:1398] T c651044a305c4168805825a53d910899 P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240501 20:01:07.905555 29852 tablet_bootstrap.cc:654] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:07.905827 29851 tablet_bootstrap.cc:492] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:07.905889 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e07220db7444ae97f294d70eda6a93" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:07.913842 29851 tablet_bootstrap.cc:654] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:07.909338 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e07220db7444ae97f294d70eda6a93" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:07.909370 30063 raft_consensus.cc:504] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.932358 29659 raft_consensus.cc:2379] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d in current term 1: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:07.933645 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e07220db7444ae97f294d70eda6a93" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:07.934496 29734 raft_consensus.cc:3037] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
W20240501 20:01:07.944465 29700 leader_election.cc:341] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:07.945070 29700 leader_election.cc:302] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:07.945598 30063 leader_election.cc:288] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:07.899585 29987 leader_election.cc:288] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:07.948102 29734 raft_consensus.cc:2454] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:07.948834 29856 raft_consensus.cc:2726] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:07.951946 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e07220db7444ae97f294d70eda6a93" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:07.952235 29624 leader_election.cc:302] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:07.953480 30063 raft_consensus.cc:2781] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:07.930403 29988 consensus_queue.cc:1035] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.025s
I20240501 20:01:07.954979 30063 raft_consensus.cc:686] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:07.956403 29914 consensus_queue.cc:1035] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:01:07.956216 30063 consensus_queue.cc:237] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.964800 29851 tablet_bootstrap.cc:492] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:07.965415 29851 ts_tablet_manager.cc:1367] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.060s user 0.018s sys 0.038s
I20240501 20:01:07.969214 29851 raft_consensus.cc:348] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.969935 29851 raft_consensus.cc:374] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:07.970221 29851 raft_consensus.cc:729] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:07.970850 29851 consensus_queue.cc:260] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.972487 29851 ts_tablet_manager.cc:1398] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240501 20:01:07.973323 29851 tablet_bootstrap.cc:492] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:07.977753 29914 raft_consensus.cc:482] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:07.978219 29914 raft_consensus.cc:504] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.979832 29557 catalog_manager.cc:5484] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 1 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:07.980556 29914 leader_election.cc:288] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:07.986467 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e9f75387564275a60d38e9a910319c" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:07.978608 29851 tablet_bootstrap.cc:654] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:07.991832 29849 tablet_bootstrap.cc:492] T 25e07220db7444ae97f294d70eda6a93 P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:07.992434 29849 ts_tablet_manager.cc:1367] T 25e07220db7444ae97f294d70eda6a93 P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.119s user 0.005s sys 0.010s
I20240501 20:01:07.994808 29849 raft_consensus.cc:348] T 25e07220db7444ae97f294d70eda6a93 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:07.996194 29849 raft_consensus.cc:374] T 25e07220db7444ae97f294d70eda6a93 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240501 20:01:07.995468 29625 leader_election.cc:341] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:07.996464 29849 raft_consensus.cc:729] T 25e07220db7444ae97f294d70eda6a93 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:08.000018 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e9f75387564275a60d38e9a910319c" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:08.003968 29698 leader_election.cc:341] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
W20240501 20:01:08.004662 29700 leader_election.cc:341] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:08.005414 29700 leader_election.cc:302] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:08.006292 29849 consensus_queue.cc:260] T 25e07220db7444ae97f294d70eda6a93 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.007257 29988 raft_consensus.cc:2726] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:08.007870 29851 tablet_bootstrap.cc:492] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:08.008370 29851 ts_tablet_manager.cc:1367] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.035s user 0.008s sys 0.004s
I20240501 20:01:08.011483 29851 raft_consensus.cc:348] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.012380 29851 raft_consensus.cc:374] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:08.012670 29851 raft_consensus.cc:729] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:08.013270 29851 consensus_queue.cc:260] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.014467 30063 raft_consensus.cc:482] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:08.014937 30063 raft_consensus.cc:504] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.017565 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd5e83faaff4e2da4e4cc22ba410d33" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:08.019490 30063 leader_election.cc:288] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:08.020633 29851 ts_tablet_manager.cc:1398] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.012s user 0.004s sys 0.001s
I20240501 20:01:08.021739 29851 tablet_bootstrap.cc:492] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:08.022661 29852 tablet_bootstrap.cc:492] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:08.023110 29852 ts_tablet_manager.cc:1367] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.125s user 0.039s sys 0.037s
I20240501 20:01:08.024864 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd5e83faaff4e2da4e4cc22ba410d33" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:08.025506 29805 raft_consensus.cc:2454] T 8bd5e83faaff4e2da4e4cc22ba410d33 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 2.
I20240501 20:01:08.025962 29849 ts_tablet_manager.cc:1398] T 25e07220db7444ae97f294d70eda6a93 P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.033s user 0.002s sys 0.003s
I20240501 20:01:08.026829 29849 tablet_bootstrap.cc:492] T a8e9f75387564275a60d38e9a910319c P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:08.026460 29852 raft_consensus.cc:348] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.026994 29851 tablet_bootstrap.cc:654] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:08.027179 29852 raft_consensus.cc:374] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:08.027482 29852 raft_consensus.cc:729] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:08.028229 29852 consensus_queue.cc:260] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.031455 29625 leader_election.cc:302] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters: fdfeac4190e045f290e003b2298d988d
I20240501 20:01:08.030804 29852 ts_tablet_manager.cc:1398] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.007s user 0.005s sys 0.004s
I20240501 20:01:08.034018 29852 tablet_bootstrap.cc:492] T 3c5345f6eb62449ba69b4a9c2e349b31 P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:08.037299 30063 raft_consensus.cc:2781] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:08.037695 30063 raft_consensus.cc:482] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:08.038195 30063 raft_consensus.cc:3037] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:08.038543 29851 tablet_bootstrap.cc:492] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:08.038996 29851 ts_tablet_manager.cc:1367] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.003s
I20240501 20:01:08.042335 29656 raft_consensus.cc:3037] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:08.045568 29851 raft_consensus.cc:348] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.046298 29851 raft_consensus.cc:374] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:08.046583 29851 raft_consensus.cc:729] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:08.047178 29851 consensus_queue.cc:260] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.054071 30063 raft_consensus.cc:504] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.056722 29851 ts_tablet_manager.cc:1398] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.017s user 0.005s sys 0.001s
I20240501 20:01:08.057452 29851 tablet_bootstrap.cc:492] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:08.070638 29849 tablet_bootstrap.cc:654] T a8e9f75387564275a60d38e9a910319c P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:08.070828 30063 leader_election.cc:288] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:08.071663 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd5e83faaff4e2da4e4cc22ba410d33" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:08.072351 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd5e83faaff4e2da4e4cc22ba410d33" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:08.075992 29852 tablet_bootstrap.cc:654] T 3c5345f6eb62449ba69b4a9c2e349b31 P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:08.082971 30065 raft_consensus.cc:482] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:08.083782 29851 tablet_bootstrap.cc:654] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:08.083474 30065 raft_consensus.cc:504] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.076860 29804 raft_consensus.cc:3037] T 8bd5e83faaff4e2da4e4cc22ba410d33 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:08.087527 30065 leader_election.cc:288] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:08.090543 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b337b8e3e38d4ad6b1076fbd0d0a1609" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:08.090865 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b337b8e3e38d4ad6b1076fbd0d0a1609" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:08.091668 29805 raft_consensus.cc:2379] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 1: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:08.092967 29625 leader_election.cc:302] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:08.094643 30063 raft_consensus.cc:3037] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:08.098317 29804 raft_consensus.cc:2454] T 8bd5e83faaff4e2da4e4cc22ba410d33 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 3.
I20240501 20:01:08.099521 29625 leader_election.cc:302] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters: fdfeac4190e045f290e003b2298d988d
I20240501 20:01:08.100554 29979 raft_consensus.cc:2781] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:01:08.101001 29979 raft_consensus.cc:686] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:08.101804 29979 consensus_queue.cc:237] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.129568 29849 tablet_bootstrap.cc:492] T a8e9f75387564275a60d38e9a910319c P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:08.132843 29849 ts_tablet_manager.cc:1367] T a8e9f75387564275a60d38e9a910319c P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.106s user 0.012s sys 0.009s
I20240501 20:01:08.133596 29851 tablet_bootstrap.cc:492] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:08.134055 29851 ts_tablet_manager.cc:1367] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.077s user 0.014s sys 0.003s
I20240501 20:01:08.135100 29557 catalog_manager.cc:5484] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 2 to 3, leader changed from fdfeac4190e045f290e003b2298d988d (127.27.217.194) to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 3 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:08.135350 29849 raft_consensus.cc:348] T a8e9f75387564275a60d38e9a910319c P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.136112 29849 raft_consensus.cc:374] T a8e9f75387564275a60d38e9a910319c P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:08.136368 29849 raft_consensus.cc:729] T a8e9f75387564275a60d38e9a910319c P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:08.136860 29851 raft_consensus.cc:348] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.137569 29851 raft_consensus.cc:374] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:08.137825 29851 raft_consensus.cc:729] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:08.137444 29849 consensus_queue.cc:260] T a8e9f75387564275a60d38e9a910319c P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.130944 30063 raft_consensus.cc:2726] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:08.139699 29849 ts_tablet_manager.cc:1398] T a8e9f75387564275a60d38e9a910319c P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20240501 20:01:08.139497 29851 consensus_queue.cc:260] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.140594 29849 tablet_bootstrap.cc:492] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:08.141963 29851 ts_tablet_manager.cc:1398] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20240501 20:01:08.142307 29852 tablet_bootstrap.cc:492] T 3c5345f6eb62449ba69b4a9c2e349b31 P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:08.142732 29852 ts_tablet_manager.cc:1367] T 3c5345f6eb62449ba69b4a9c2e349b31 P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.109s user 0.007s sys 0.006s
I20240501 20:01:08.142773 29851 tablet_bootstrap.cc:492] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:08.148622 29852 raft_consensus.cc:348] T 3c5345f6eb62449ba69b4a9c2e349b31 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.154728 29849 tablet_bootstrap.cc:654] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:08.154757 30063 consensus_queue.cc:1035] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:01:08.157020 29851 tablet_bootstrap.cc:654] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:08.149430 29852 raft_consensus.cc:374] T 3c5345f6eb62449ba69b4a9c2e349b31 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:08.163462 29852 raft_consensus.cc:729] T 3c5345f6eb62449ba69b4a9c2e349b31 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:08.165023 29852 consensus_queue.cc:260] T 3c5345f6eb62449ba69b4a9c2e349b31 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.169602 29852 ts_tablet_manager.cc:1398] T 3c5345f6eb62449ba69b4a9c2e349b31 P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.026s user 0.005s sys 0.001s
I20240501 20:01:08.172204 29852 tablet_bootstrap.cc:492] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:08.180856 29909 consensus_queue.cc:1035] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:01:08.184002 29805 raft_consensus.cc:1226] T 8bd5e83faaff4e2da4e4cc22ba410d33 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 2. Current term is 3. Ops: []
I20240501 20:01:08.194880 29987 raft_consensus.cc:482] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:08.195811 29987 raft_consensus.cc:504] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.200654 29909 consensus_queue.cc:1046] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20240501 20:01:08.207460 29909 raft_consensus.cc:3032] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:01:08.208024 29909 raft_consensus.cc:729] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:08.209250 29909 consensus_queue.cc:260] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.217267 29909 raft_consensus.cc:3037] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:08.211148 29896 raft_consensus.cc:482] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:08.220388 29896 raft_consensus.cc:504] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.223098 29896 leader_election.cc:288] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:08.214825 29851 tablet_bootstrap.cc:492] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:08.212069 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5345f6eb62449ba69b4a9c2e349b31" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:08.224925 29851 ts_tablet_manager.cc:1367] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.082s user 0.016s sys 0.039s
I20240501 20:01:08.238561 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c651044a305c4168805825a53d910899" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:08.239318 29734 raft_consensus.cc:2454] T c651044a305c4168805825a53d910899 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 0.
I20240501 20:01:08.224321 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c651044a305c4168805825a53d910899" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:08.240017 29851 raft_consensus.cc:348] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.240957 29851 raft_consensus.cc:374] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:08.241286 29851 raft_consensus.cc:729] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
W20240501 20:01:08.242324 29772 leader_election.cc:341] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:08.242076 29851 consensus_queue.cc:260] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.243038 29773 leader_election.cc:302] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:08.245595 29656 raft_consensus.cc:2454] T 3c5345f6eb62449ba69b4a9c2e349b31 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 0.
I20240501 20:01:08.249559 29698 leader_election.cc:302] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:08.250378 29988 raft_consensus.cc:2781] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:08.250773 29988 raft_consensus.cc:482] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:08.251096 29988 raft_consensus.cc:3037] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:08.252606 29985 raft_consensus.cc:2781] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:08.253070 29985 raft_consensus.cc:482] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:08.253464 29985 raft_consensus.cc:3037] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:08.204133 29987 leader_election.cc:288] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:08.261484 29849 tablet_bootstrap.cc:492] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:08.262140 29849 ts_tablet_manager.cc:1367] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.122s user 0.028s sys 0.008s
I20240501 20:01:08.264703 29849 raft_consensus.cc:348] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.265241 29849 raft_consensus.cc:374] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:08.265511 29849 raft_consensus.cc:729] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:08.265818 29851 ts_tablet_manager.cc:1398] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.034s user 0.003s sys 0.003s
I20240501 20:01:08.269179 29851 tablet_bootstrap.cc:492] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:08.281525 29849 consensus_queue.cc:260] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.262064 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5345f6eb62449ba69b4a9c2e349b31" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:08.289886 29852 tablet_bootstrap.cc:654] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:08.289698 29988 raft_consensus.cc:504] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.283103 30075 raft_consensus.cc:482] T a8e9f75387564275a60d38e9a910319c P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:08.300697 30075 raft_consensus.cc:504] T a8e9f75387564275a60d38e9a910319c P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.302799 30075 leader_election.cc:288] T a8e9f75387564275a60d38e9a910319c P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:08.306890 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e9f75387564275a60d38e9a910319c" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:08.307530 29734 raft_consensus.cc:2454] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 0.
I20240501 20:01:08.303968 30075 raft_consensus.cc:482] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:08.309583 29773 leader_election.cc:302] T a8e9f75387564275a60d38e9a910319c P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:08.309607 30075 raft_consensus.cc:504] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.313469 30075 leader_election.cc:288] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:08.314147 30075 raft_consensus.cc:2781] T a8e9f75387564275a60d38e9a910319c P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:08.314587 30075 raft_consensus.cc:482] T a8e9f75387564275a60d38e9a910319c P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:08.315018 30075 raft_consensus.cc:3037] T a8e9f75387564275a60d38e9a910319c P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:08.317550 29896 raft_consensus.cc:482] T 25e07220db7444ae97f294d70eda6a93 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240501 20:01:08.318543 29700 leader_election.cc:341] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:08.318259 29896 raft_consensus.cc:504] T 25e07220db7444ae97f294d70eda6a93 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.319386 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211e27a3b0084b89a6dd7550849b5a65" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:08.320765 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211e27a3b0084b89a6dd7550849b5a65" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:08.325592 29734 raft_consensus.cc:2454] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 0.
I20240501 20:01:08.325789 29985 raft_consensus.cc:504] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.326004 29654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e07220db7444ae97f294d70eda6a93" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:08.326683 29987 raft_consensus.cc:482] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:08.332408 29987 raft_consensus.cc:504] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.335011 29987 leader_election.cc:288] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:08.327009 29909 consensus_queue.cc:1035] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240501 20:01:08.326423 29655 raft_consensus.cc:2454] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 0.
I20240501 20:01:08.329352 29896 leader_election.cc:288] T 25e07220db7444ae97f294d70eda6a93 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:08.337353 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5525f546426c47a38edf7f56ad04b7ab" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:08.329286 29773 leader_election.cc:302] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:08.345351 29896 raft_consensus.cc:2781] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:08.345909 29896 raft_consensus.cc:482] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:08.347087 29896 raft_consensus.cc:3037] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:08.351399 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e07220db7444ae97f294d70eda6a93" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:08.352406 29734 raft_consensus.cc:2379] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 1: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:08.360514 29909 raft_consensus.cc:482] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:08.361677 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e9f75387564275a60d38e9a910319c" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:08.363265 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5345f6eb62449ba69b4a9c2e349b31" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:08.364187 29656 raft_consensus.cc:3037] T 3c5345f6eb62449ba69b4a9c2e349b31 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
W20240501 20:01:08.364868 29772 leader_election.cc:341] T a8e9f75387564275a60d38e9a910319c P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
W20240501 20:01:08.365432 29700 leader_election.cc:341] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:08.361711 29909 raft_consensus.cc:504] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.370787 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5525f546426c47a38edf7f56ad04b7ab" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:08.374469 29849 ts_tablet_manager.cc:1398] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.112s user 0.009s sys 0.003s
I20240501 20:01:08.339190 29988 leader_election.cc:288] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:08.382606 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5345f6eb62449ba69b4a9c2e349b31" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:08.390851 29985 leader_election.cc:288] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:08.375648 30065 consensus_queue.cc:1035] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
W20240501 20:01:08.396320 29772 consensus_peers.cc:485] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d -> Peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Couldn't send request to peer 4c8da1d580614449a636b855bf11b95f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:08.380172 29909 leader_election.cc:288] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:08.407099 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c651044a305c4168805825a53d910899" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:08.412299 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48fd0a8e289401aa7417c42ff2c81b2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:08.413798 30075 raft_consensus.cc:504] T a8e9f75387564275a60d38e9a910319c P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.415076 29851 tablet_bootstrap.cc:654] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
W20240501 20:01:08.434281 29698 leader_election.cc:341] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:08.449208 29852 tablet_bootstrap.cc:492] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:08.449702 29852 ts_tablet_manager.cc:1367] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.278s user 0.020s sys 0.024s
I20240501 20:01:08.453295 29988 raft_consensus.cc:482] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:08.456030 29988 raft_consensus.cc:504] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.457966 29988 leader_election.cc:288] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:08.462543 29772 leader_election.cc:341] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:08.463691 29896 raft_consensus.cc:504] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.464938 29803 raft_consensus.cc:3037] T 25e07220db7444ae97f294d70eda6a93 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:08.436744 29698 leader_election.cc:302] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:08.472016 29849 tablet_bootstrap.cc:492] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:08.473438 30075 leader_election.cc:288] T a8e9f75387564275a60d38e9a910319c P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:08.482620 29852 raft_consensus.cc:348] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.483379 29852 raft_consensus.cc:374] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:08.483639 29852 raft_consensus.cc:729] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:08.487025 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211e27a3b0084b89a6dd7550849b5a65" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:08.487890 29660 raft_consensus.cc:3037] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:08.489363 29852 consensus_queue.cc:260] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.441083 29773 leader_election.cc:302] T 25e07220db7444ae97f294d70eda6a93 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:08.494851 29914 raft_consensus.cc:2726] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:08.491536 29852 ts_tablet_manager.cc:1398] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.042s user 0.003s sys 0.002s
I20240501 20:01:08.497725 29852 tablet_bootstrap.cc:492] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:08.468132 29896 leader_election.cc:288] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:08.508790 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48fd0a8e289401aa7417c42ff2c81b2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:08.507864 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e9f75387564275a60d38e9a910319c" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
W20240501 20:01:08.516243 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 81659.8us
I20240501 20:01:08.528481 29851 tablet_bootstrap.cc:492] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:08.529013 29851 ts_tablet_manager.cc:1367] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.262s user 0.011s sys 0.018s
I20240501 20:01:08.532023 29851 raft_consensus.cc:348] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.532627 29851 raft_consensus.cc:374] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:08.532950 29851 raft_consensus.cc:729] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:08.533948 29851 consensus_queue.cc:260] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.524348 29656 raft_consensus.cc:2454] T 3c5345f6eb62449ba69b4a9c2e349b31 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
I20240501 20:01:08.532347 30073 raft_consensus.cc:482] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:08.542673 30073 raft_consensus.cc:504] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.546236 29851 ts_tablet_manager.cc:1398] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20240501 20:01:08.533437 29660 raft_consensus.cc:2454] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
I20240501 20:01:08.553529 30073 leader_election.cc:288] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
W20240501 20:01:08.591415 29772 leader_election.cc:341] T a8e9f75387564275a60d38e9a910319c P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:08.592752 29772 leader_election.cc:302] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:08.593910 30116 raft_consensus.cc:2781] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:08.594650 30116 raft_consensus.cc:686] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:08.595495 30116 consensus_queue.cc:237] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.596247 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece8d4ed84c34b748f7b6272b31c90ac" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
W20240501 20:01:08.598374 29772 leader_election.cc:341] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:08.599506 30115 consensus_queue.cc:1035] T 010ceabf3e5a4b1c9330b80dade765bb P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:01:08.647444 30114 raft_consensus.cc:482] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:08.647902 30114 raft_consensus.cc:504] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.651364 30114 leader_election.cc:288] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:08.654639 29805 raft_consensus.cc:1261] T 8bd5e83faaff4e2da4e4cc22ba410d33 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240501 20:01:08.655926 30114 consensus_queue.cc:1035] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:08.658911 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5525fc4b82a44b0991c3dd6098fde81" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:08.659577 29735 raft_consensus.cc:2454] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 0.
I20240501 20:01:08.660506 29896 raft_consensus.cc:2726] T 25e07220db7444ae97f294d70eda6a93 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:08.662576 29624 leader_election.cc:302] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:08.660279 30114 consensus_queue.cc:1035] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:08.663902 30065 consensus_queue.cc:1035] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:08.664496 30111 raft_consensus.cc:2781] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:08.664897 30111 raft_consensus.cc:482] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:08.665225 30111 raft_consensus.cc:3037] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:08.664281 29849 tablet_bootstrap.cc:654] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:08.676589 30111 raft_consensus.cc:504] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.688918 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5525fc4b82a44b0991c3dd6098fde81" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:08.691635 29733 raft_consensus.cc:3037] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:08.704424 29852 tablet_bootstrap.cc:654] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:08.715910 29914 raft_consensus.cc:482] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240501 20:01:08.719535 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 285283us
I20240501 20:01:08.712635 30111 leader_election.cc:288] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:08.721135 29733 raft_consensus.cc:2454] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:08.723938 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5525fc4b82a44b0991c3dd6098fde81" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:08.727473 29849 tablet_bootstrap.cc:492] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:08.727875 29849 ts_tablet_manager.cc:1367] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.256s user 0.012s sys 0.015s
I20240501 20:01:08.724826 29660 raft_consensus.cc:2379] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d in current term 1: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:08.730201 29624 leader_election.cc:302] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:08.734287 30111 raft_consensus.cc:2781] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:08.734817 30111 raft_consensus.cc:686] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:08.735736 30111 consensus_queue.cc:237] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.731096 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5525fc4b82a44b0991c3dd6098fde81" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:08.740368 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5525fc4b82a44b0991c3dd6098fde81" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:08.716799 29914 raft_consensus.cc:504] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.743723 30117 raft_consensus.cc:482] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:08.743845 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c651044a305c4168805825a53d910899" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
W20240501 20:01:08.744115 29625 leader_election.cc:341] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:08.744496 29731 raft_consensus.cc:3037] T c651044a305c4168805825a53d910899 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
W20240501 20:01:08.745178 29625 leader_election.cc:341] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:08.744343 30117 raft_consensus.cc:504] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.755262 29732 raft_consensus.cc:1261] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20240501 20:01:08.756359 29732 pending_rounds.cc:76] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d: Aborting all ops after (but not including) 1
I20240501 20:01:08.756701 29732 pending_rounds.cc:97] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20240501 20:01:08.756525 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e9f75387564275a60d38e9a910319c" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:08.757097 29732 raft_consensus.cc:2866] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 20:01:08.757361 29730 raft_consensus.cc:3037] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:08.760102 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211e27a3b0084b89a6dd7550849b5a65" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:08.760782 29734 raft_consensus.cc:3037] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:08.770846 29852 tablet_bootstrap.cc:492] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:08.771407 29852 ts_tablet_manager.cc:1367] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.274s user 0.016s sys 0.022s
I20240501 20:01:08.770499 29977 raft_consensus.cc:482] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:08.777246 30117 leader_election.cc:288] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:08.784700 29851 tablet_bootstrap.cc:492] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:08.785212 29977 raft_consensus.cc:504] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:08.466665 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 76400.3us
I20240501 20:01:08.788959 29977 leader_election.cc:288] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:08.789042 29700 leader_election.cc:341] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
W20240501 20:01:08.789768 29700 leader_election.cc:341] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:08.788280 30117 consensus_queue.cc:1035] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20240501 20:01:08.791041 29914 leader_election.cc:288] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:08.800098 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5525fc4b82a44b0991c3dd6098fde81" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:08.799507 29698 leader_election.cc:341] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:08.801541 29698 leader_election.cc:302] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:08.804808 29698 leader_election.cc:302] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:08.808944 29989 raft_consensus.cc:2781] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:08.809451 29989 raft_consensus.cc:686] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:08.810326 29989 consensus_queue.cc:237] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.807863 30119 raft_consensus.cc:2726] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:08.822561 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1a08b455b94ae089396a15c837ddc2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:08.824857 29852 raft_consensus.cc:348] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.829772 29852 raft_consensus.cc:374] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240501 20:01:08.830364 29606 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240501 20:01:08.830471 29852 raft_consensus.cc:729] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:08.831928 29852 consensus_queue.cc:260] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.834379 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a2bba62f434d1c8489b55c8fec2187" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:08.837479 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1a08b455b94ae089396a15c837ddc2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:08.838238 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6797d82dda2948d8a7445fcd33e44752" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:08.839635 29896 consensus_queue.cc:1035] T 010ceabf3e5a4b1c9330b80dade765bb P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:08.841915 29849 raft_consensus.cc:348] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.842523 29849 raft_consensus.cc:374] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:08.842741 29849 raft_consensus.cc:729] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:08.843221 29849 consensus_queue.cc:260] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:08.841691 29625 leader_election.cc:341] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
W20240501 20:01:08.845165 29698 leader_election.cc:341] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:08.840013 29909 raft_consensus.cc:482] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:08.846776 29852 ts_tablet_manager.cc:1398] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.075s user 0.007s sys 0.000s
I20240501 20:01:08.839239 29985 consensus_queue.cc:1035] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20240501 20:01:08.849702 29909 raft_consensus.cc:504] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:08.857554 29625 leader_election.cc:341] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:08.860998 29909 leader_election.cc:288] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:08.851857 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a2bba62f434d1c8489b55c8fec2187" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
W20240501 20:01:08.861215 29700 leader_election.cc:341] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
W20240501 20:01:08.873214 29624 leader_election.cc:341] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:08.873745 29624 leader_election.cc:302] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:08.881225 29849 ts_tablet_manager.cc:1398] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.153s user 0.005s sys 0.001s
I20240501 20:01:08.887157 29849 tablet_bootstrap.cc:492] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:08.854408 29731 raft_consensus.cc:2454] T c651044a305c4168805825a53d910899 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
I20240501 20:01:08.867842 29700 leader_election.cc:302] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:08.879681 29979 raft_consensus.cc:2726] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:08.931608 29730 raft_consensus.cc:2454] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
W20240501 20:01:08.927639 29624 leader_election.cc:341] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:08.931713 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece8d4ed84c34b748f7b6272b31c90ac" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:08.940418 29734 raft_consensus.cc:2454] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
I20240501 20:01:08.941512 29659 raft_consensus.cc:2454] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 0.
I20240501 20:01:08.929924 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6797d82dda2948d8a7445fcd33e44752" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:08.936234 29988 raft_consensus.cc:2726] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240501 20:01:08.947619 30131 log.cc:924] Time spent T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 0.098s user 0.000s sys 0.001s
I20240501 20:01:08.950177 29698 leader_election.cc:302] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:08.937433 29624 leader_election.cc:302] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:08.951035 29977 raft_consensus.cc:2726] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:08.950274 29773 leader_election.cc:302] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:08.972831 29851 tablet_bootstrap.cc:654] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
W20240501 20:01:08.945219 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 83045.8us
I20240501 20:01:08.973908 29989 raft_consensus.cc:2781] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:08.979038 29989 raft_consensus.cc:482] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:08.979351 29989 raft_consensus.cc:3037] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:08.978399 29896 raft_consensus.cc:2781] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:08.994805 29989 raft_consensus.cc:504] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:08.997710 29773 leader_election.cc:302] T a8e9f75387564275a60d38e9a910319c P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:08.998265 29979 raft_consensus.cc:482] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:08.999341 29985 raft_consensus.cc:2781] T a8e9f75387564275a60d38e9a910319c P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:08.999804 29985 raft_consensus.cc:686] T a8e9f75387564275a60d38e9a910319c P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:08.999591 29979 raft_consensus.cc:504] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:08.999965 29700 leader_election.cc:341] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:09.000746 29985 consensus_queue.cc:237] T a8e9f75387564275a60d38e9a910319c P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:09.001479 29700 leader_election.cc:302] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:08.985090 29896 raft_consensus.cc:686] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:09.004931 29987 raft_consensus.cc:2726] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:09.005779 29896 consensus_queue.cc:237] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:09.006947 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece8d4ed84c34b748f7b6272b31c90ac" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:09.007613 29808 raft_consensus.cc:2454] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 0.
I20240501 20:01:09.010818 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0374bb790ed345ed90452595f71383a2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:09.012990 29625 leader_election.cc:341] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:09.004434 29979 leader_election.cc:288] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:09.023883 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece8d4ed84c34b748f7b6272b31c90ac" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:09.024931 29729 raft_consensus.cc:2379] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 1: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:09.062870 29773 leader_election.cc:302] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:09.064371 30115 raft_consensus.cc:3037] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:09.066560 29558 catalog_manager.cc:5484] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 1 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:09.068812 29852 tablet_bootstrap.cc:492] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:09.067905 29849 tablet_bootstrap.cc:654] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:09.075686 29989 leader_election.cc:288] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:09.076287 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece8d4ed84c34b748f7b6272b31c90ac" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:09.076952 29659 raft_consensus.cc:3037] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:09.096746 29659 raft_consensus.cc:2454] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
I20240501 20:01:09.119344 29698 leader_election.cc:302] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:09.118672 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0374bb790ed345ed90452595f71383a2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:09.121282 30121 raft_consensus.cc:2781] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:09.122350 30121 raft_consensus.cc:686] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:09.123152 30121 consensus_queue.cc:237] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:09.162170 29624 leader_election.cc:341] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:09.162689 29624 leader_election.cc:302] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:09.163403 30117 raft_consensus.cc:2726] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:09.163538 29851 tablet_bootstrap.cc:492] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:09.164021 29851 ts_tablet_manager.cc:1367] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.380s user 0.038s sys 0.036s
I20240501 20:01:09.166365 29851 raft_consensus.cc:348] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:09.166915 29851 raft_consensus.cc:374] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:09.156970 29896 raft_consensus.cc:482] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:09.167182 29851 raft_consensus.cc:729] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:09.168498 29851 consensus_queue.cc:260] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:09.169942 29851 ts_tablet_manager.cc:1398] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20240501 20:01:09.170699 29851 tablet_bootstrap.cc:492] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:09.172586 29896 raft_consensus.cc:504] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:09.174921 29896 leader_election.cc:288] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:09.176402 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48fd0a8e289401aa7417c42ff2c81b2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
W20240501 20:01:09.179921 29772 leader_election.cc:341] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:09.179613 30115 raft_consensus.cc:2726] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:09.183295 29852 tablet_bootstrap.cc:654] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:09.183357 30116 consensus_queue.cc:1035] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:01:09.184979 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece8d4ed84c34b748f7b6272b31c90ac" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:09.191984 29896 consensus_queue.cc:1035] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:09.193420 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48fd0a8e289401aa7417c42ff2c81b2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:09.194197 29729 raft_consensus.cc:2454] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 0.
I20240501 20:01:09.195262 29773 leader_election.cc:302] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:09.196955 30075 raft_consensus.cc:2781] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:09.213053 29851 tablet_bootstrap.cc:654] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:09.205632 30075 raft_consensus.cc:482] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:09.215867 29849 tablet_bootstrap.cc:492] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:09.216755 29849 ts_tablet_manager.cc:1367] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.330s user 0.010s sys 0.008s
I20240501 20:01:09.215054 30075 raft_consensus.cc:3037] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:09.242929 29809 raft_consensus.cc:2454] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
I20240501 20:01:09.249090 29849 raft_consensus.cc:348] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:09.249836 29849 raft_consensus.cc:374] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:09.250101 29849 raft_consensus.cc:729] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:09.250725 29849 consensus_queue.cc:260] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:09.254063 30121 raft_consensus.cc:482] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:09.254549 30121 raft_consensus.cc:504] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:09.255388 29851 tablet_bootstrap.cc:492] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:09.254916 30075 raft_consensus.cc:504] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:09.256016 29851 ts_tablet_manager.cc:1367] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.085s user 0.009s sys 0.001s
I20240501 20:01:09.259330 29849 ts_tablet_manager.cc:1398] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.042s user 0.004s sys 0.002s
I20240501 20:01:09.257230 30121 leader_election.cc:288] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:09.259975 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec073fe645614fe28e7837de3c22510b" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:09.260182 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48fd0a8e289401aa7417c42ff2c81b2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:09.261596 29852 tablet_bootstrap.cc:492] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
W20240501 20:01:09.261803 29698 leader_election.cc:341] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:09.262048 29852 ts_tablet_manager.cc:1367] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.193s user 0.007s sys 0.007s
W20240501 20:01:09.261845 29772 leader_election.cc:341] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:09.259799 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec073fe645614fe28e7837de3c22510b" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:09.264236 29700 leader_election.cc:341] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:09.264530 29849 tablet_bootstrap.cc:492] T 3c5345f6eb62449ba69b4a9c2e349b31 P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:09.264678 29700 leader_election.cc:302] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:09.266139 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48fd0a8e289401aa7417c42ff2c81b2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:09.266753 29733 raft_consensus.cc:3037] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:09.268136 30121 raft_consensus.cc:2726] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:09.259480 29851 raft_consensus.cc:348] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:09.268384 29852 raft_consensus.cc:348] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:09.269219 29851 raft_consensus.cc:374] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:09.269354 29852 raft_consensus.cc:374] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:09.269557 29851 raft_consensus.cc:729] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:09.269791 29852 raft_consensus.cc:729] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:09.271498 29851 consensus_queue.cc:260] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:09.275465 29851 ts_tablet_manager.cc:1398] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.019s user 0.003s sys 0.004s
I20240501 20:01:09.275547 29733 raft_consensus.cc:2454] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
I20240501 20:01:09.276513 29851 tablet_bootstrap.cc:492] T fa7e89c0572e4ee7be8e5315b459bdb3 P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:09.277880 29773 leader_election.cc:302] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:09.278832 29896 raft_consensus.cc:2781] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:09.279270 29896 raft_consensus.cc:686] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:09.280122 29896 consensus_queue.cc:237] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:09.282310 30075 leader_election.cc:288] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
W20240501 20:01:09.283521 29700 consensus_peers.cc:485] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d -> Peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Couldn't send request to peer ee0d97247d394a8facfa774f15f3a67d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:09.270823 29852 consensus_queue.cc:260] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:09.286528 29852 ts_tablet_manager.cc:1398] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.024s user 0.008s sys 0.000s
I20240501 20:01:09.287245 29852 tablet_bootstrap.cc:492] T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:09.291224 29849 tablet_bootstrap.cc:654] T 3c5345f6eb62449ba69b4a9c2e349b31 P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:09.291268 29851 tablet_bootstrap.cc:654] T fa7e89c0572e4ee7be8e5315b459bdb3 P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:09.293519 29557 catalog_manager.cc:5484] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 0 to 1, leader changed from <none> to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 1 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:09.297821 29852 tablet_bootstrap.cc:654] T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:09.305635 29851 tablet_bootstrap.cc:492] T fa7e89c0572e4ee7be8e5315b459bdb3 P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:09.306095 29851 ts_tablet_manager.cc:1367] T fa7e89c0572e4ee7be8e5315b459bdb3 P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.030s user 0.010s sys 0.002s
I20240501 20:01:09.309002 29851 raft_consensus.cc:348] T fa7e89c0572e4ee7be8e5315b459bdb3 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:09.310060 29851 raft_consensus.cc:374] T fa7e89c0572e4ee7be8e5315b459bdb3 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:09.310465 29851 raft_consensus.cc:729] T fa7e89c0572e4ee7be8e5315b459bdb3 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:09.311324 29851 consensus_queue.cc:260] T fa7e89c0572e4ee7be8e5315b459bdb3 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:09.312263 30121 raft_consensus.cc:482] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:09.312774 30121 raft_consensus.cc:504] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:09.314783 29851 ts_tablet_manager.cc:1398] T fa7e89c0572e4ee7be8e5315b459bdb3 P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20240501 20:01:09.314962 29809 raft_consensus.cc:3037] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:09.315256 30121 leader_election.cc:288] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:09.315894 29851 tablet_bootstrap.cc:492] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:09.317390 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a2bba62f434d1c8489b55c8fec2187" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:09.320562 29700 leader_election.cc:341] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:09.319635 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a2bba62f434d1c8489b55c8fec2187" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:09.321558 29659 raft_consensus.cc:2454] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 0.
I20240501 20:01:09.322635 29851 tablet_bootstrap.cc:654] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:09.322691 29698 leader_election.cc:302] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:09.323594 30121 raft_consensus.cc:2781] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:09.323997 30121 raft_consensus.cc:482] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:09.324389 30121 raft_consensus.cc:3037] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:09.325474 29849 tablet_bootstrap.cc:492] T 3c5345f6eb62449ba69b4a9c2e349b31 P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:09.325938 29849 ts_tablet_manager.cc:1367] T 3c5345f6eb62449ba69b4a9c2e349b31 P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.062s user 0.003s sys 0.009s
I20240501 20:01:09.332484 29849 raft_consensus.cc:348] T 3c5345f6eb62449ba69b4a9c2e349b31 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:09.333463 29849 raft_consensus.cc:374] T 3c5345f6eb62449ba69b4a9c2e349b31 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:09.333798 29849 raft_consensus.cc:729] T 3c5345f6eb62449ba69b4a9c2e349b31 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:09.336026 29979 consensus_queue.cc:1035] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:01:09.334589 29849 consensus_queue.cc:260] T 3c5345f6eb62449ba69b4a9c2e349b31 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:09.345721 29849 ts_tablet_manager.cc:1398] T 3c5345f6eb62449ba69b4a9c2e349b31 P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.019s user 0.006s sys 0.002s
I20240501 20:01:09.346714 29849 tablet_bootstrap.cc:492] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:09.349004 29556 catalog_manager.cc:5484] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 0 to 1, leader changed from <none> to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 1 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:09.356838 29852 tablet_bootstrap.cc:492] T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:09.359123 29852 ts_tablet_manager.cc:1367] T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.072s user 0.013s sys 0.006s
I20240501 20:01:09.367113 30117 consensus_queue.cc:1035] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:01:09.370496 29851 tablet_bootstrap.cc:492] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:09.370301 29852 raft_consensus.cc:348] T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:09.370995 29851 ts_tablet_manager.cc:1367] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.055s user 0.025s sys 0.010s
I20240501 20:01:09.371095 29852 raft_consensus.cc:374] T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:09.371412 29852 raft_consensus.cc:729] T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:09.371882 30121 raft_consensus.cc:504] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:09.373433 29852 consensus_queue.cc:260] T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:09.377633 29849 tablet_bootstrap.cc:654] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:09.377549 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a2bba62f434d1c8489b55c8fec2187" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:09.378212 29851 raft_consensus.cc:348] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:09.378931 29851 raft_consensus.cc:374] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:09.379173 29851 raft_consensus.cc:729] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:09.379673 29851 consensus_queue.cc:260] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:09.378521 29659 raft_consensus.cc:3037] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:09.384059 29851 ts_tablet_manager.cc:1398] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.013s user 0.004s sys 0.003s
I20240501 20:01:09.385272 29557 catalog_manager.cc:5484] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 0 to 1, leader changed from <none> to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 1 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:09.387616 29557 catalog_manager.cc:5484] T a8e9f75387564275a60d38e9a910319c P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 0 to 1, leader changed from <none> to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 1 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:09.389101 29851 tablet_bootstrap.cc:492] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:09.389416 29557 catalog_manager.cc:5484] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 0 to 1, leader changed from <none> to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 1 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:09.400331 30121 leader_election.cc:288] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:09.422868 29852 ts_tablet_manager.cc:1398] T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.063s user 0.004s sys 0.002s
I20240501 20:01:09.424309 29852 tablet_bootstrap.cc:492] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:09.404174 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a2bba62f434d1c8489b55c8fec2187" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:09.436501 29659 raft_consensus.cc:2454] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
I20240501 20:01:09.437526 29698 leader_election.cc:302] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:09.439703 30121 raft_consensus.cc:2781] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:09.456777 30121 raft_consensus.cc:686] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:09.457718 29849 tablet_bootstrap.cc:492] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:09.458163 29849 ts_tablet_manager.cc:1367] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.112s user 0.015s sys 0.010s
I20240501 20:01:09.439329 30139 consensus_queue.cc:1035] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
W20240501 20:01:09.453739 29700 leader_election.cc:341] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:09.468031 29851 tablet_bootstrap.cc:654] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:09.468097 29849 raft_consensus.cc:348] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:09.469020 29849 raft_consensus.cc:374] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:09.469264 29849 raft_consensus.cc:729] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:09.469738 29849 consensus_queue.cc:260] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:09.462181 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 55196us
I20240501 20:01:09.480608 29849 ts_tablet_manager.cc:1398] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.022s user 0.005s sys 0.000s
I20240501 20:01:09.482614 29849 tablet_bootstrap.cc:492] T 36057b3f24d44b9aac3a1378053a395f P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:09.485265 29556 catalog_manager.cc:5484] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 0 to 1, leader changed from <none> to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 1 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:09.458074 30121 consensus_queue.cc:237] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:09.491075 29852 tablet_bootstrap.cc:654] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:09.492075 29896 consensus_queue.cc:1035] T a8e9f75387564275a60d38e9a910319c P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:01:09.514815 29659 raft_consensus.cc:3037] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:09.525588 29851 tablet_bootstrap.cc:492] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:09.526084 29851 ts_tablet_manager.cc:1367] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.137s user 0.022s sys 0.024s
I20240501 20:01:09.530966 29851 raft_consensus.cc:348] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:09.531502 29851 raft_consensus.cc:374] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:09.531736 29851 raft_consensus.cc:729] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:09.532719 29852 tablet_bootstrap.cc:492] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:09.532325 29851 consensus_queue.cc:260] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:09.533124 29852 ts_tablet_manager.cc:1367] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.109s user 0.012s sys 0.005s
I20240501 20:01:09.534494 29851 ts_tablet_manager.cc:1398] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240501 20:01:09.535133 29852 raft_consensus.cc:348] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:09.535468 29851 tablet_bootstrap.cc:492] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:09.535616 29852 raft_consensus.cc:374] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:09.535913 29852 raft_consensus.cc:729] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:09.536696 29849 tablet_bootstrap.cc:654] T 36057b3f24d44b9aac3a1378053a395f P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:09.536512 29852 consensus_queue.cc:260] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:09.538597 29852 ts_tablet_manager.cc:1398] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240501 20:01:09.539245 29852 tablet_bootstrap.cc:492] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:09.547782 29656 raft_consensus.cc:3037] T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:09.551548 29852 tablet_bootstrap.cc:654] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:09.552661 29851 tablet_bootstrap.cc:654] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
W20240501 20:01:09.564366 29681 debug-util.cc:398] Leaking SignalData structure 0x7b08000d25e0 after lost signal to thread 28522
W20240501 20:01:09.565380 29681 debug-util.cc:398] Leaking SignalData structure 0x7b080009db60 after lost signal to thread 29592
W20240501 20:01:09.566437 29681 debug-util.cc:398] Leaking SignalData structure 0x7b080027a800 after lost signal to thread 29684
W20240501 20:01:09.567396 29681 debug-util.cc:398] Leaking SignalData structure 0x7b0800276880 after lost signal to thread 29759
I20240501 20:01:09.572798 30075 consensus_queue.cc:1035] T a8e9f75387564275a60d38e9a910319c P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20240501 20:01:09.577332 29681 debug-util.cc:398] Leaking SignalData structure 0x7b080022bec0 after lost signal to thread 29833
I20240501 20:01:09.592895 30119 consensus_queue.cc:1035] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:09.602313 29554 catalog_manager.cc:5484] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 0 to 1, leader changed from <none> to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 1 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:09.609709 29849 tablet_bootstrap.cc:492] T 36057b3f24d44b9aac3a1378053a395f P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:09.611413 29849 ts_tablet_manager.cc:1367] T 36057b3f24d44b9aac3a1378053a395f P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.130s user 0.026s sys 0.010s
I20240501 20:01:09.622346 29852 tablet_bootstrap.cc:492] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:09.622846 29852 ts_tablet_manager.cc:1367] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.084s user 0.018s sys 0.007s
I20240501 20:01:09.623903 29660 raft_consensus.cc:3037] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:09.625903 29852 raft_consensus.cc:348] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:09.627645 29852 raft_consensus.cc:374] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:09.628101 29852 raft_consensus.cc:729] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:09.620429 29849 raft_consensus.cc:348] T 36057b3f24d44b9aac3a1378053a395f P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:09.629302 29852 consensus_queue.cc:260] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:09.630250 29849 raft_consensus.cc:374] T 36057b3f24d44b9aac3a1378053a395f P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:09.630551 29849 raft_consensus.cc:729] T 36057b3f24d44b9aac3a1378053a395f P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:09.631119 29849 consensus_queue.cc:260] T 36057b3f24d44b9aac3a1378053a395f P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:09.633419 29849 ts_tablet_manager.cc:1398] T 36057b3f24d44b9aac3a1378053a395f P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20240501 20:01:09.638862 29849 tablet_bootstrap.cc:492] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:09.640667 30075 consensus_queue.cc:1035] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:01:09.647472 29852 ts_tablet_manager.cc:1398] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.024s user 0.007s sys 0.000s
I20240501 20:01:09.648506 29852 tablet_bootstrap.cc:492] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:09.668207 29896 raft_consensus.cc:482] T 36057b3f24d44b9aac3a1378053a395f P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:09.668776 29896 raft_consensus.cc:504] T 36057b3f24d44b9aac3a1378053a395f P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:09.671554 29896 leader_election.cc:288] T 36057b3f24d44b9aac3a1378053a395f P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:09.673360 29851 tablet_bootstrap.cc:492] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:09.673146 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36057b3f24d44b9aac3a1378053a395f" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:09.673812 29851 ts_tablet_manager.cc:1367] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.139s user 0.036s sys 0.025s
I20240501 20:01:09.676604 30119 raft_consensus.cc:482] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240501 20:01:09.678056 29772 leader_election.cc:341] T 36057b3f24d44b9aac3a1378053a395f P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:09.676934 29851 raft_consensus.cc:348] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:09.676126 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36057b3f24d44b9aac3a1378053a395f" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:09.678656 29851 raft_consensus.cc:374] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:09.679522 29851 raft_consensus.cc:729] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:09.677881 30119 raft_consensus.cc:504] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:09.680228 29733 raft_consensus.cc:2454] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 0.
I20240501 20:01:09.680142 29851 consensus_queue.cc:260] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:09.682863 30119 leader_election.cc:288] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:09.684762 29851 ts_tablet_manager.cc:1398] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20240501 20:01:09.684844 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0374bb790ed345ed90452595f71383a2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:09.685286 29773 leader_election.cc:302] T 36057b3f24d44b9aac3a1378053a395f P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:09.685766 29851 tablet_bootstrap.cc:492] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
W20240501 20:01:09.686179 29700 leader_election.cc:341] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:09.686951 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0374bb790ed345ed90452595f71383a2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:09.687546 29659 raft_consensus.cc:2454] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 0.
I20240501 20:01:09.688613 29698 leader_election.cc:302] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:09.688642 29896 raft_consensus.cc:2781] T 36057b3f24d44b9aac3a1378053a395f P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:09.689028 29896 raft_consensus.cc:482] T 36057b3f24d44b9aac3a1378053a395f P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:09.689335 29896 raft_consensus.cc:3037] T 36057b3f24d44b9aac3a1378053a395f P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:09.690788 30140 raft_consensus.cc:2781] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:09.691236 30140 raft_consensus.cc:482] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:09.691529 30140 raft_consensus.cc:3037] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:09.698117 29896 raft_consensus.cc:504] T 36057b3f24d44b9aac3a1378053a395f P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:09.699707 29896 leader_election.cc:288] T 36057b3f24d44b9aac3a1378053a395f P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:09.709049 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36057b3f24d44b9aac3a1378053a395f" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:09.710826 29914 consensus_queue.cc:1035] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:09.717634 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36057b3f24d44b9aac3a1378053a395f" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:09.722254 30075 raft_consensus.cc:482] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:09.722510 29734 raft_consensus.cc:3037] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:09.722770 30075 raft_consensus.cc:504] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:09.735751 29852 tablet_bootstrap.cc:654] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:09.736506 30075 leader_election.cc:288] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
W20240501 20:01:09.721112 29772 leader_election.cc:341] T 36057b3f24d44b9aac3a1378053a395f P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:09.740432 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5525f546426c47a38edf7f56ad04b7ab" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:09.746505 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5525f546426c47a38edf7f56ad04b7ab" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
W20240501 20:01:09.796532 29772 leader_election.cc:341] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
W20240501 20:01:09.797724 30179 log.cc:924] Time spent T c651044a305c4168805825a53d910899 P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
I20240501 20:01:09.798375 30075 consensus_queue.cc:1035] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:09.801841 29851 tablet_bootstrap.cc:654] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:09.790249 29732 raft_consensus.cc:2454] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 0.
I20240501 20:01:09.791782 29734 raft_consensus.cc:2454] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
I20240501 20:01:09.806370 30075 raft_consensus.cc:482] T 3c5345f6eb62449ba69b4a9c2e349b31 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:09.816447 30075 raft_consensus.cc:504] T 3c5345f6eb62449ba69b4a9c2e349b31 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:09.817134 30140 raft_consensus.cc:504] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:09.822487 29852 tablet_bootstrap.cc:492] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:09.822896 29852 ts_tablet_manager.cc:1367] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.175s user 0.014s sys 0.000s
I20240501 20:01:09.833882 29849 tablet_bootstrap.cc:654] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:09.828619 30121 raft_consensus.cc:482] T fa7e89c0572e4ee7be8e5315b459bdb3 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:09.967676 29852 raft_consensus.cc:348] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:09.968392 29852 raft_consensus.cc:374] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:09.968621 29852 raft_consensus.cc:729] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:09.840274 29808 raft_consensus.cc:3037] T 3c5345f6eb62449ba69b4a9c2e349b31 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:09.887228 30075 leader_election.cc:288] T 3c5345f6eb62449ba69b4a9c2e349b31 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:09.911778 30140 leader_election.cc:288] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:09.973428 29852 consensus_queue.cc:260] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:09.979281 29852 ts_tablet_manager.cc:1398] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.156s user 0.008s sys 0.000s
I20240501 20:01:09.980321 29852 tablet_bootstrap.cc:492] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:09.983676 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5345f6eb62449ba69b4a9c2e349b31" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:09.868500 29773 leader_election.cc:302] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:09.960824 30121 raft_consensus.cc:504] T fa7e89c0572e4ee7be8e5315b459bdb3 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:09.997759 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0374bb790ed345ed90452595f71383a2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:10.010919 29657 raft_consensus.cc:3037] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:10.009990 30187 raft_consensus.cc:2781] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:10.038836 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0374bb790ed345ed90452595f71383a2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:10.029840 29653 raft_consensus.cc:3037] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
W20240501 20:01:10.016363 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 146687us
I20240501 20:01:10.046772 29773 leader_election.cc:302] T 36057b3f24d44b9aac3a1378053a395f P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:10.024967 30187 raft_consensus.cc:482] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:10.047798 30187 raft_consensus.cc:3037] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:10.048410 30073 raft_consensus.cc:2781] T 36057b3f24d44b9aac3a1378053a395f P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader election won for term 1
W20240501 20:01:10.056030 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 57281.2us
I20240501 20:01:10.054867 30121 leader_election.cc:288] T fa7e89c0572e4ee7be8e5315b459bdb3 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:10.057314 30073 raft_consensus.cc:686] T 36057b3f24d44b9aac3a1378053a395f P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:10.060864 30073 consensus_queue.cc:237] T 36057b3f24d44b9aac3a1378053a395f P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:10.068442 30186 log.cc:924] Time spent T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20240501 20:01:10.084573 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e89c0572e4ee7be8e5315b459bdb3" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:10.085884 30119 raft_consensus.cc:482] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:10.086380 30119 raft_consensus.cc:504] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.088286 30119 leader_election.cc:288] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:10.092835 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:10.094352 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e89c0572e4ee7be8e5315b459bdb3" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:10.078766 29849 tablet_bootstrap.cc:492] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:10.095544 29849 ts_tablet_manager.cc:1367] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.457s user 0.020s sys 0.031s
I20240501 20:01:10.102167 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5345f6eb62449ba69b4a9c2e349b31" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:10.104112 29849 raft_consensus.cc:348] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.104971 29849 raft_consensus.cc:374] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:10.105893 29849 raft_consensus.cc:729] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:10.107038 29849 consensus_queue.cc:260] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.109968 29849 ts_tablet_manager.cc:1398] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.014s user 0.001s sys 0.006s
I20240501 20:01:10.110206 30115 consensus_queue.cc:1035] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:10.112164 29849 tablet_bootstrap.cc:492] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:10.105592 29852 tablet_bootstrap.cc:654] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
W20240501 20:01:10.115796 29700 leader_election.cc:341] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:10.114840 30166 raft_consensus.cc:482] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240501 20:01:10.119475 29698 leader_election.cc:341] T fa7e89c0572e4ee7be8e5315b459bdb3 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:10.118698 30166 raft_consensus.cc:504] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:10.121734 29700 leader_election.cc:341] T fa7e89c0572e4ee7be8e5315b459bdb3 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:10.123054 29700 leader_election.cc:302] T fa7e89c0572e4ee7be8e5315b459bdb3 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:10.123430 30166 leader_election.cc:288] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:10.124409 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48fd0a8e289401aa7417c42ff2c81b2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:10.127828 29851 tablet_bootstrap.cc:492] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:10.128309 29851 ts_tablet_manager.cc:1367] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.443s user 0.030s sys 0.012s
I20240501 20:01:10.128449 29657 raft_consensus.cc:2454] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
W20240501 20:01:10.129838 29698 leader_election.cc:341] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:10.129269 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:10.132305 29698 leader_election.cc:302] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:10.133730 30119 raft_consensus.cc:2726] T fa7e89c0572e4ee7be8e5315b459bdb3 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:10.134130 29851 raft_consensus.cc:348] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.134865 29851 raft_consensus.cc:374] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:10.135126 29851 raft_consensus.cc:729] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:10.134335 29773 leader_election.cc:302] T 3c5345f6eb62449ba69b4a9c2e349b31 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:10.135730 29851 consensus_queue.cc:260] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:10.136629 29700 consensus_peers.cc:485] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d -> Peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Couldn't send request to peer ee0d97247d394a8facfa774f15f3a67d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:10.140630 29851 ts_tablet_manager.cc:1398] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20240501 20:01:10.134357 30119 raft_consensus.cc:2781] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:10.141458 29851 tablet_bootstrap.cc:492] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
W20240501 20:01:10.142397 29700 leader_election.cc:341] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:10.144910 29700 leader_election.cc:302] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:10.148937 30119 raft_consensus.cc:686] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:10.154644 30184 consensus_queue.cc:1035] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240501 20:01:10.157019 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48fd0a8e289401aa7417c42ff2c81b2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:10.157788 29732 raft_consensus.cc:2454] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:10.160593 29985 consensus_queue.cc:1035] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:10.162724 30119 consensus_queue.cc:237] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.163309 30115 raft_consensus.cc:2726] T 3c5345f6eb62449ba69b4a9c2e349b31 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:10.164605 29624 leader_election.cc:302] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:10.149914 29989 raft_consensus.cc:2726] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:10.196228 30187 raft_consensus.cc:504] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.198618 30187 leader_election.cc:288] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:10.198917 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5525f546426c47a38edf7f56ad04b7ab" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:10.199577 29657 raft_consensus.cc:3037] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:10.212024 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5525f546426c47a38edf7f56ad04b7ab" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:10.212709 29732 raft_consensus.cc:3037] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
W20240501 20:01:10.220947 29606 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240501 20:01:10.226621 29849 tablet_bootstrap.cc:654] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:10.228269 29851 tablet_bootstrap.cc:654] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:10.240281 30166 raft_consensus.cc:2781] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:10.242367 29852 tablet_bootstrap.cc:492] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:10.242833 29852 ts_tablet_manager.cc:1367] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.263s user 0.019s sys 0.002s
I20240501 20:01:10.217329 29557 catalog_manager.cc:5484] T 36057b3f24d44b9aac3a1378053a395f P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 0 to 1, leader changed from <none> to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 1 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:10.242105 29914 raft_consensus.cc:482] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:10.246282 29914 raft_consensus.cc:504] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.248726 29914 leader_election.cc:288] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:10.250521 29852 raft_consensus.cc:348] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.251253 29852 raft_consensus.cc:374] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:10.251523 29852 raft_consensus.cc:729] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:10.252099 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f5d1686214485d91a0baf4b870740e" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:10.252640 30166 raft_consensus.cc:482] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:10.253331 30166 raft_consensus.cc:3037] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:10.254765 29852 consensus_queue.cc:260] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.271819 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f5d1686214485d91a0baf4b870740e" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
W20240501 20:01:10.273541 29698 leader_election.cc:341] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:10.244402 29732 raft_consensus.cc:2454] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
W20240501 20:01:10.275976 29700 leader_election.cc:341] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:10.281942 29852 ts_tablet_manager.cc:1398] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.039s user 0.007s sys 0.000s
I20240501 20:01:10.281773 29700 leader_election.cc:302] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:10.286909 29852 tablet_bootstrap.cc:492] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:10.283018 29556 catalog_manager.cc:5484] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 0 to 1, leader changed from <none> to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 1 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:10.269291 29657 raft_consensus.cc:2454] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
I20240501 20:01:10.294411 30182 raft_consensus.cc:2726] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:10.295349 30168 raft_consensus.cc:482] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:10.295246 29772 leader_election.cc:302] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:10.295866 30168 raft_consensus.cc:504] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.297796 30168 leader_election.cc:288] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:10.300662 29985 raft_consensus.cc:2781] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:10.301182 29985 raft_consensus.cc:686] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:10.301674 30166 raft_consensus.cc:504] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.301985 29985 consensus_queue.cc:237] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.304003 30166 leader_election.cc:288] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:10.307108 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48fd0a8e289401aa7417c42ff2c81b2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:10.308813 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8edcd903224d48f889b46e1b5318e0ec" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:10.310179 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48fd0a8e289401aa7417c42ff2c81b2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
W20240501 20:01:10.312880 29625 leader_election.cc:341] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:10.314332 29625 leader_election.cc:302] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:10.312922 29653 raft_consensus.cc:1226] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:10.317380 30168 raft_consensus.cc:2726] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 20:01:10.323746 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8edcd903224d48f889b46e1b5318e0ec" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:10.325243 30073 consensus_queue.cc:1046] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:10.326103 30073 raft_consensus.cc:3032] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:10.326421 30073 raft_consensus.cc:729] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
W20240501 20:01:10.326860 29624 leader_election.cc:341] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:10.327281 29624 leader_election.cc:302] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:10.327165 30073 consensus_queue.cc:260] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.328691 30073 raft_consensus.cc:3037] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:10.329026 30166 raft_consensus.cc:2726] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:10.333680 29849 tablet_bootstrap.cc:492] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:10.334425 29849 ts_tablet_manager.cc:1367] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.222s user 0.005s sys 0.009s
I20240501 20:01:10.336877 29849 raft_consensus.cc:348] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.337565 29849 raft_consensus.cc:374] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:10.337824 29849 raft_consensus.cc:729] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:10.341975 29851 tablet_bootstrap.cc:492] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:10.342087 30115 raft_consensus.cc:482] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:10.342710 29851 ts_tablet_manager.cc:1367] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.201s user 0.006s sys 0.013s
I20240501 20:01:10.342778 30115 raft_consensus.cc:504] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.344938 29852 tablet_bootstrap.cc:654] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:10.345153 30115 leader_election.cc:288] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:10.345358 29851 raft_consensus.cc:348] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.346486 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6797d82dda2948d8a7445fcd33e44752" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:10.346575 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6797d82dda2948d8a7445fcd33e44752" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:10.346530 29851 raft_consensus.cc:374] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:10.350988 29851 raft_consensus.cc:729] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:10.341747 29849 consensus_queue.cc:260] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:10.348505 29772 leader_election.cc:341] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:10.351711 29851 consensus_queue.cc:260] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.351130 29732 raft_consensus.cc:2454] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 0.
I20240501 20:01:10.359308 29851 ts_tablet_manager.cc:1398] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.016s user 0.012s sys 0.000s
I20240501 20:01:10.359364 29849 ts_tablet_manager.cc:1398] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.025s user 0.005s sys 0.001s
I20240501 20:01:10.360388 29851 tablet_bootstrap.cc:492] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:10.360745 29849 tablet_bootstrap.cc:492] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:10.360574 29773 leader_election.cc:302] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:10.370569 29985 raft_consensus.cc:2781] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:10.371598 29985 raft_consensus.cc:482] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:10.372500 29985 raft_consensus.cc:3037] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:10.375846 29849 tablet_bootstrap.cc:654] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:10.379724 29851 tablet_bootstrap.cc:654] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:10.381719 29985 raft_consensus.cc:504] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.384907 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6797d82dda2948d8a7445fcd33e44752" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
W20240501 20:01:10.387640 29772 leader_election.cc:341] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:10.389132 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6797d82dda2948d8a7445fcd33e44752" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:10.389614 29732 raft_consensus.cc:3037] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:10.392113 29852 tablet_bootstrap.cc:492] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:10.392472 29852 ts_tablet_manager.cc:1367] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.106s user 0.009s sys 0.012s
I20240501 20:01:10.389766 29985 leader_election.cc:288] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:10.400375 29732 raft_consensus.cc:2454] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
I20240501 20:01:10.400466 29852 raft_consensus.cc:348] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.404244 29852 raft_consensus.cc:374] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:10.404548 29852 raft_consensus.cc:729] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:10.404917 29773 leader_election.cc:302] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:10.405139 29852 consensus_queue.cc:260] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.406283 29985 raft_consensus.cc:2781] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:10.406793 29985 raft_consensus.cc:686] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:10.407159 29852 ts_tablet_manager.cc:1398] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.014s user 0.004s sys 0.000s
I20240501 20:01:10.407644 29985 consensus_queue.cc:237] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.408481 29852 tablet_bootstrap.cc:492] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:10.415979 29852 tablet_bootstrap.cc:654] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:10.434602 29554 catalog_manager.cc:5484] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 0 to 1, leader changed from <none> to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 1 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:10.462491 29554 catalog_manager.cc:5484] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 0 to 1, leader changed from <none> to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 1 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:10.463063 30073 consensus_queue.cc:1035] T 36057b3f24d44b9aac3a1378053a395f P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:10.465925 29851 tablet_bootstrap.cc:492] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:10.466580 29851 ts_tablet_manager.cc:1367] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.107s user 0.025s sys 0.014s
I20240501 20:01:10.469157 29851 raft_consensus.cc:348] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.469816 29851 raft_consensus.cc:374] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:10.470078 29851 raft_consensus.cc:729] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:10.471808 29851 consensus_queue.cc:260] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.479805 29851 ts_tablet_manager.cc:1398] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.013s user 0.008s sys 0.000s
I20240501 20:01:10.481524 29851 tablet_bootstrap.cc:492] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:10.486982 29849 tablet_bootstrap.cc:492] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:10.487416 29852 tablet_bootstrap.cc:492] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:10.487579 29849 ts_tablet_manager.cc:1367] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.127s user 0.013s sys 0.024s
I20240501 20:01:10.487871 29852 ts_tablet_manager.cc:1367] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.080s user 0.016s sys 0.009s
I20240501 20:01:10.492079 29849 raft_consensus.cc:348] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.492781 29849 raft_consensus.cc:374] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:10.493023 29849 raft_consensus.cc:729] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:10.493557 29849 consensus_queue.cc:260] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.495033 29849 ts_tablet_manager.cc:1398] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240501 20:01:10.500546 29852 raft_consensus.cc:348] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.501255 29852 raft_consensus.cc:374] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:10.501518 29852 raft_consensus.cc:729] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
W20240501 20:01:10.501958 29772 consensus_peers.cc:485] T 36057b3f24d44b9aac3a1378053a395f P ee0d97247d394a8facfa774f15f3a67d -> Peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Couldn't send request to peer 4c8da1d580614449a636b855bf11b95f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:10.502117 29852 consensus_queue.cc:260] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.504227 29852 ts_tablet_manager.cc:1398] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20240501 20:01:10.505230 29849 tablet_bootstrap.cc:492] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:10.510278 30073 raft_consensus.cc:482] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:10.511188 29849 tablet_bootstrap.cc:654] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:10.510809 30073 raft_consensus.cc:504] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.512888 30073 leader_election.cc:288] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:10.517915 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f5d1686214485d91a0baf4b870740e" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:10.527573 29734 raft_consensus.cc:2454] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 0.
I20240501 20:01:10.521268 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f5d1686214485d91a0baf4b870740e" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:10.530297 29773 leader_election.cc:302] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
W20240501 20:01:10.532150 29772 leader_election.cc:341] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:10.532200 30115 raft_consensus.cc:2781] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:10.533130 30115 raft_consensus.cc:482] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:10.533504 30115 raft_consensus.cc:3037] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:10.553628 30168 raft_consensus.cc:482] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:10.554395 30168 raft_consensus.cc:504] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.558830 30168 leader_election.cc:288] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:10.560498 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6797d82dda2948d8a7445fcd33e44752" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:10.562827 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6797d82dda2948d8a7445fcd33e44752" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:10.563634 29730 raft_consensus.cc:2379] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 1: Already voted for candidate ee0d97247d394a8facfa774f15f3a67d in this term.
I20240501 20:01:10.564714 29624 leader_election.cc:302] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:10.565832 30168 raft_consensus.cc:3037] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:10.567029 30119 raft_consensus.cc:482] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:10.568765 30119 raft_consensus.cc:504] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.571787 30119 leader_election.cc:288] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:10.571990 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8cd40cb4044d18a86001ce6ddf97ab" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
W20240501 20:01:10.573794 29698 leader_election.cc:341] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:10.576511 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8cd40cb4044d18a86001ce6ddf97ab" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:10.577778 29700 leader_election.cc:341] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:10.578207 29700 leader_election.cc:302] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:10.579205 30119 raft_consensus.cc:2726] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:10.584684 30115 raft_consensus.cc:504] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.587504 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f5d1686214485d91a0baf4b870740e" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
W20240501 20:01:10.589018 29772 leader_election.cc:341] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:10.621981 29851 tablet_bootstrap.cc:654] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:10.622480 29849 tablet_bootstrap.cc:492] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:10.622927 29849 ts_tablet_manager.cc:1367] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.118s user 0.037s sys 0.050s
I20240501 20:01:10.625248 29849 raft_consensus.cc:348] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.625910 29849 raft_consensus.cc:374] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:10.626240 29849 raft_consensus.cc:729] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:10.626799 29849 consensus_queue.cc:260] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.628360 29849 ts_tablet_manager.cc:1398] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20240501 20:01:10.629184 29849 tablet_bootstrap.cc:492] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:10.629223 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f5d1686214485d91a0baf4b870740e" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:10.629914 29730 raft_consensus.cc:3037] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:10.636296 29852 tablet_bootstrap.cc:492] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:10.637436 29730 raft_consensus.cc:2454] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
I20240501 20:01:10.638592 29773 leader_election.cc:302] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:10.639524 30073 raft_consensus.cc:2781] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:10.640014 30073 raft_consensus.cc:686] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:10.640903 30073 consensus_queue.cc:237] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.644528 29849 tablet_bootstrap.cc:654] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:10.646150 29809 raft_consensus.cc:3037] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:10.655876 29849 tablet_bootstrap.cc:492] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:10.656383 29849 ts_tablet_manager.cc:1367] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.027s user 0.013s sys 0.004s
I20240501 20:01:10.657799 29989 consensus_queue.cc:1035] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:10.658841 29849 raft_consensus.cc:348] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.659600 29849 raft_consensus.cc:374] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:10.659852 29849 raft_consensus.cc:729] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:10.660475 29849 consensus_queue.cc:260] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.661859 30168 raft_consensus.cc:2726] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:10.665023 29852 tablet_bootstrap.cc:654] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:10.665905 29849 ts_tablet_manager.cc:1398] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.009s user 0.005s sys 0.003s
I20240501 20:01:10.666918 29849 tablet_bootstrap.cc:492] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:10.667584 30115 leader_election.cc:288] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:10.686059 29849 tablet_bootstrap.cc:654] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:10.690729 29852 tablet_bootstrap.cc:492] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:10.691321 29852 ts_tablet_manager.cc:1367] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.055s user 0.011s sys 0.002s
I20240501 20:01:10.694800 29852 raft_consensus.cc:348] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.695713 29852 raft_consensus.cc:374] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:10.696516 29852 raft_consensus.cc:729] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:10.697329 29852 consensus_queue.cc:260] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.700232 29852 ts_tablet_manager.cc:1398] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.009s user 0.008s sys 0.002s
I20240501 20:01:10.706149 29852 tablet_bootstrap.cc:492] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:10.720621 29554 catalog_manager.cc:5484] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 0 to 1, leader changed from <none> to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 1 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:10.722970 29851 tablet_bootstrap.cc:492] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:10.723769 29851 ts_tablet_manager.cc:1367] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.242s user 0.020s sys 0.000s
I20240501 20:01:10.727298 29852 tablet_bootstrap.cc:654] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
W20240501 20:01:10.735824 29700 consensus_peers.cc:485] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d -> Peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Couldn't send request to peer ee0d97247d394a8facfa774f15f3a67d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:10.751896 29849 tablet_bootstrap.cc:492] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:10.752821 29849 ts_tablet_manager.cc:1367] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.086s user 0.019s sys 0.020s
I20240501 20:01:10.751987 29851 raft_consensus.cc:348] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.753902 29851 raft_consensus.cc:374] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:10.756145 29851 raft_consensus.cc:729] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:10.756340 29849 raft_consensus.cc:348] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.757124 29849 raft_consensus.cc:374] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:10.757400 29849 raft_consensus.cc:729] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:10.758086 29849 consensus_queue.cc:260] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.757493 29851 consensus_queue.cc:260] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.760479 29849 ts_tablet_manager.cc:1398] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240501 20:01:10.761616 29849 tablet_bootstrap.cc:492] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:10.762217 29851 ts_tablet_manager.cc:1398] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.038s user 0.006s sys 0.000s
I20240501 20:01:10.763152 29851 tablet_bootstrap.cc:492] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:10.768370 29849 tablet_bootstrap.cc:654] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:10.774492 29852 tablet_bootstrap.cc:492] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:10.775106 29852 ts_tablet_manager.cc:1367] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.069s user 0.018s sys 0.007s
I20240501 20:01:10.778393 29852 raft_consensus.cc:348] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.779330 29852 raft_consensus.cc:374] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:10.779678 29852 raft_consensus.cc:729] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:10.780476 29852 consensus_queue.cc:260] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.781848 29851 tablet_bootstrap.cc:654] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:10.787606 29852 ts_tablet_manager.cc:1398] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20240501 20:01:10.788468 29852 tablet_bootstrap.cc:492] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
W20240501 20:01:10.791234 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 60451.7us
I20240501 20:01:10.792443 29851 tablet_bootstrap.cc:492] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:10.792886 29851 ts_tablet_manager.cc:1367] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.030s user 0.012s sys 0.001s
I20240501 20:01:10.793329 29849 tablet_bootstrap.cc:492] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:10.793743 29852 tablet_bootstrap.cc:654] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:10.793802 29849 ts_tablet_manager.cc:1367] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.032s user 0.019s sys 0.008s
I20240501 20:01:10.796581 29849 raft_consensus.cc:348] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.795483 30119 raft_consensus.cc:482] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:10.803040 29849 raft_consensus.cc:374] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:10.803400 29849 raft_consensus.cc:729] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:10.803426 30119 raft_consensus.cc:504] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.804214 29849 consensus_queue.cc:260] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.805986 30140 raft_consensus.cc:482] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:10.806695 30119 leader_election.cc:288] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:10.806452 30140 raft_consensus.cc:504] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.807092 29849 ts_tablet_manager.cc:1398] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.013s user 0.004s sys 0.003s
I20240501 20:01:10.808107 29849 tablet_bootstrap.cc:492] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:10.795161 29851 raft_consensus.cc:348] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.808295 30140 leader_election.cc:288] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:10.808641 29851 raft_consensus.cc:374] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:10.808918 29851 raft_consensus.cc:729] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:10.809512 29851 consensus_queue.cc:260] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.811647 29851 ts_tablet_manager.cc:1398] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.018s user 0.003s sys 0.000s
I20240501 20:01:10.812705 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec073fe645614fe28e7837de3c22510b" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:10.814857 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8edcd903224d48f889b46e1b5318e0ec" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:10.815433 29653 raft_consensus.cc:2454] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 0.
I20240501 20:01:10.816586 30073 raft_consensus.cc:482] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:10.817055 30073 raft_consensus.cc:504] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.818928 30073 leader_election.cc:288] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:10.820842 29849 tablet_bootstrap.cc:654] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:10.821578 29851 tablet_bootstrap.cc:492] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:10.825481 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec073fe645614fe28e7837de3c22510b" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:10.832706 30115 consensus_queue.cc:1035] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:10.835704 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8edcd903224d48f889b46e1b5318e0ec" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:10.825655 29698 leader_election.cc:341] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
W20240501 20:01:10.841114 29700 leader_election.cc:341] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:10.843456 29698 leader_election.cc:302] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
W20240501 20:01:10.848690 29700 leader_election.cc:341] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:10.849673 30166 raft_consensus.cc:482] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:10.850322 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:10.850318 30166 raft_consensus.cc:504] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.854851 30166 leader_election.cc:288] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:10.855801 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:10.856266 29657 raft_consensus.cc:2454] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 0.
I20240501 20:01:10.856583 29809 raft_consensus.cc:2454] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 0.
I20240501 20:01:10.857642 29772 leader_election.cc:302] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:10.857795 29625 leader_election.cc:302] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:10.858474 30115 raft_consensus.cc:2781] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:10.858541 30166 raft_consensus.cc:2781] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:10.858963 30115 raft_consensus.cc:482] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:10.859445 30115 raft_consensus.cc:3037] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:10.859542 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:10.860267 29730 raft_consensus.cc:2454] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 0.
I20240501 20:01:10.860980 29851 tablet_bootstrap.cc:654] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:10.858990 30166 raft_consensus.cc:482] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:10.863807 30166 raft_consensus.cc:3037] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:10.862999 30119 raft_consensus.cc:482] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:10.864742 30119 raft_consensus.cc:504] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.866645 30119 leader_election.cc:288] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:10.866817 30115 raft_consensus.cc:504] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.868192 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:10.869856 30073 raft_consensus.cc:482] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:10.870110 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:10.870357 30073 raft_consensus.cc:504] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.870815 30166 raft_consensus.cc:504] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.872018 29657 raft_consensus.cc:2379] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 1: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:10.872295 30073 leader_election.cc:288] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
W20240501 20:01:10.872519 29700 leader_election.cc:341] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:10.869850 30115 leader_election.cc:288] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:10.873420 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:10.874087 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1a08b455b94ae089396a15c837ddc2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:10.874404 29730 raft_consensus.cc:3037] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:10.874752 29653 raft_consensus.cc:2454] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 0.
I20240501 20:01:10.877466 30119 raft_consensus.cc:2726] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240501 20:01:10.877629 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 52007.4us
I20240501 20:01:10.878373 29698 leader_election.cc:302] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:10.879101 30119 raft_consensus.cc:2781] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:10.879410 29730 raft_consensus.cc:2454] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:10.879546 30119 raft_consensus.cc:482] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:10.880031 30119 raft_consensus.cc:3037] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:10.881654 29772 leader_election.cc:302] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:10.884384 30115 raft_consensus.cc:2781] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:10.884740 30115 raft_consensus.cc:482] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:10.885082 30115 raft_consensus.cc:3037] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:10.886384 30119 raft_consensus.cc:504] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.888334 30119 leader_election.cc:288] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:10.890333 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8edcd903224d48f889b46e1b5318e0ec" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:10.890903 30115 raft_consensus.cc:504] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:10.892377 29700 leader_election.cc:341] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:10.894369 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1a08b455b94ae089396a15c837ddc2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:10.894984 29653 raft_consensus.cc:3037] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:10.896164 30115 leader_election.cc:288] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:10.897480 30166 leader_election.cc:288] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:10.897800 29852 tablet_bootstrap.cc:492] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:10.898298 29852 ts_tablet_manager.cc:1367] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.110s user 0.018s sys 0.019s
I20240501 20:01:10.900676 29852 raft_consensus.cc:348] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.901324 29852 raft_consensus.cc:374] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:10.901590 29852 raft_consensus.cc:729] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:10.902222 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:10.902984 29809 raft_consensus.cc:2379] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 1: Already voted for candidate ee0d97247d394a8facfa774f15f3a67d in this term.
I20240501 20:01:10.903839 29852 consensus_queue.cc:260] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.906193 30166 raft_consensus.cc:482] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:10.906633 30166 raft_consensus.cc:504] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.908492 30166 leader_election.cc:288] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:10.912513 29852 ts_tablet_manager.cc:1398] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20240501 20:01:10.913089 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e89c0572e4ee7be8e5315b459bdb3" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:10.916388 29809 raft_consensus.cc:2454] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 0.
I20240501 20:01:10.921741 29625 leader_election.cc:302] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:10.924954 29849 tablet_bootstrap.cc:492] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:10.925395 29849 ts_tablet_manager.cc:1367] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.117s user 0.017s sys 0.004s
I20240501 20:01:10.927605 29849 raft_consensus.cc:348] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.930275 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e89c0572e4ee7be8e5315b459bdb3" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:10.930934 29732 raft_consensus.cc:2454] T fa7e89c0572e4ee7be8e5315b459bdb3 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 0.
I20240501 20:01:10.932091 29849 raft_consensus.cc:374] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:10.932751 29849 raft_consensus.cc:729] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:10.933804 29849 consensus_queue.cc:260] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.936678 29849 ts_tablet_manager.cc:1398] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20240501 20:01:10.937528 29624 leader_election.cc:302] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:10.939563 30166 raft_consensus.cc:2781] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:10.944223 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8edcd903224d48f889b46e1b5318e0ec" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:10.944662 30166 raft_consensus.cc:482] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:10.944906 29660 raft_consensus.cc:3037] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:10.945078 30166 raft_consensus.cc:3037] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:10.951316 30168 raft_consensus.cc:2781] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:10.951982 30168 raft_consensus.cc:686] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:10.953737 30168 consensus_queue.cc:237] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.940284 29852 tablet_bootstrap.cc:492] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:10.924631 29653 raft_consensus.cc:2454] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
I20240501 20:01:10.942180 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:10.955624 29657 raft_consensus.cc:2454] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 0.
I20240501 20:01:10.957372 29772 leader_election.cc:302] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:10.956681 30166 raft_consensus.cc:504] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.958281 29985 raft_consensus.cc:2781] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:10.958803 29985 raft_consensus.cc:686] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:10.959630 29985 consensus_queue.cc:237] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.959765 30166 leader_election.cc:288] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:10.963040 29852 tablet_bootstrap.cc:654] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:10.964195 29851 tablet_bootstrap.cc:492] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:10.964810 29851 ts_tablet_manager.cc:1367] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.143s user 0.013s sys 0.003s
I20240501 20:01:10.966878 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e89c0572e4ee7be8e5315b459bdb3" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:10.967541 29734 raft_consensus.cc:3037] T fa7e89c0572e4ee7be8e5315b459bdb3 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:10.971762 29851 raft_consensus.cc:348] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.974339 29851 raft_consensus.cc:374] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:10.977058 29851 raft_consensus.cc:729] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:10.977993 29851 consensus_queue.cc:260] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.987551 29660 raft_consensus.cc:2454] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
I20240501 20:01:10.988229 29851 ts_tablet_manager.cc:1398] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.023s user 0.007s sys 0.000s
I20240501 20:01:10.990087 29852 tablet_bootstrap.cc:492] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:10.990530 29852 ts_tablet_manager.cc:1367] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.050s user 0.015s sys 0.007s
I20240501 20:01:10.990782 29851 tablet_bootstrap.cc:492] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:10.995405 29852 raft_consensus.cc:348] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:10.957608 29849 tablet_bootstrap.cc:492] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:10.996107 29852 raft_consensus.cc:374] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:10.996417 29852 raft_consensus.cc:729] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:10.996749 29851 tablet_bootstrap.cc:654] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:11.001785 30140 raft_consensus.cc:482] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:11.002280 30140 raft_consensus.cc:504] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:11.004447 30140 leader_election.cc:288] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:11.005393 29852 consensus_queue.cc:260] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:11.006963 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:11.007514 29852 ts_tablet_manager.cc:1398] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.017s user 0.005s sys 0.000s
I20240501 20:01:11.007740 29730 raft_consensus.cc:2379] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 1: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:11.008560 29734 raft_consensus.cc:2454] T fa7e89c0572e4ee7be8e5315b459bdb3 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:11.008723 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1a08b455b94ae089396a15c837ddc2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:11.009083 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:11.009866 29732 raft_consensus.cc:2379] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 1: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:11.010861 29851 tablet_bootstrap.cc:492] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:11.010936 29624 leader_election.cc:302] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:11.011440 29851 ts_tablet_manager.cc:1367] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.021s user 0.012s sys 0.005s
I20240501 20:01:11.011816 30168 raft_consensus.cc:2781] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:11.012544 30168 raft_consensus.cc:686] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:11.013363 30168 consensus_queue.cc:237] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:11.014148 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e89c0572e4ee7be8e5315b459bdb3" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:11.014000 29851 raft_consensus.cc:348] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:11.014874 29809 raft_consensus.cc:3037] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:11.015051 29851 raft_consensus.cc:374] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:11.015483 29851 raft_consensus.cc:729] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:11.016222 29851 consensus_queue.cc:260] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:11.014317 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1a08b455b94ae089396a15c837ddc2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:11.017282 29728 raft_consensus.cc:2454] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 0.
I20240501 20:01:11.008896 29852 tablet_bootstrap.cc:492] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:11.018486 29851 ts_tablet_manager.cc:1398] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
I20240501 20:01:11.019703 29851 tablet_bootstrap.cc:492] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:11.023813 29852 tablet_bootstrap.cc:654] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:11.025830 29851 tablet_bootstrap.cc:654] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:11.029116 29809 raft_consensus.cc:2454] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:11.037567 29852 tablet_bootstrap.cc:492] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:11.037564 29851 tablet_bootstrap.cc:492] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:11.038246 29851 ts_tablet_manager.cc:1367] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.004s
I20240501 20:01:11.038241 29852 ts_tablet_manager.cc:1367] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.004s
I20240501 20:01:11.040839 29851 raft_consensus.cc:348] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:11.041569 29851 raft_consensus.cc:374] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:11.041301 29852 raft_consensus.cc:348] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:11.041914 29851 raft_consensus.cc:729] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:11.042335 29852 raft_consensus.cc:374] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:11.042676 29852 raft_consensus.cc:729] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:11.042670 29851 consensus_queue.cc:260] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:11.043514 29852 consensus_queue.cc:260] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:11.044914 29851 ts_tablet_manager.cc:1398] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20240501 20:01:11.045809 29851 tablet_bootstrap.cc:492] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:11.047333 29985 consensus_queue.cc:1035] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:11.048889 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1a08b455b94ae089396a15c837ddc2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:11.049491 29728 raft_consensus.cc:3037] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:11.052304 29851 tablet_bootstrap.cc:654] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:11.056376 29852 ts_tablet_manager.cc:1398] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20240501 20:01:11.057096 29728 raft_consensus.cc:2454] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
I20240501 20:01:11.057413 29852 tablet_bootstrap.cc:492] T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:11.062811 29852 tablet_bootstrap.cc:654] T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:11.056906 29554 catalog_manager.cc:5484] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 1 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:11.070887 29851 tablet_bootstrap.cc:492] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:11.071436 29851 ts_tablet_manager.cc:1367] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.013s
I20240501 20:01:11.074532 29851 raft_consensus.cc:348] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:11.074977 30187 raft_consensus.cc:482] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:11.075215 29851 raft_consensus.cc:374] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:11.075518 29851 raft_consensus.cc:729] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:11.075480 30187 raft_consensus.cc:504] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:11.076246 29851 consensus_queue.cc:260] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:11.076820 29849 tablet_bootstrap.cc:654] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:11.079681 29556 catalog_manager.cc:5484] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 0 to 1, leader changed from <none> to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 1 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:11.082077 30187 leader_election.cc:288] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:11.093686 29851 ts_tablet_manager.cc:1398] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.022s user 0.003s sys 0.005s
I20240501 20:01:11.094929 29851 tablet_bootstrap.cc:492] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:11.096230 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:11.096858 29734 raft_consensus.cc:2454] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 0.
I20240501 20:01:11.091542 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:11.098340 29652 raft_consensus.cc:2454] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 0.
I20240501 20:01:11.101743 29772 leader_election.cc:302] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:11.105386 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1a08b455b94ae089396a15c837ddc2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:11.106113 29660 raft_consensus.cc:2379] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d in current term 1: Already voted for candidate ee0d97247d394a8facfa774f15f3a67d in this term.
I20240501 20:01:11.106252 30187 raft_consensus.cc:2781] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:11.106743 30187 raft_consensus.cc:482] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:11.107177 30187 raft_consensus.cc:3037] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:11.107582 29773 leader_election.cc:302] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:11.109208 30115 consensus_queue.cc:1035] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:11.110414 30115 raft_consensus.cc:2726] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 20:01:11.104311 29557 catalog_manager.cc:5484] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 1 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:11.118750 29698 leader_election.cc:302] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:11.120234 29698 leader_election.cc:302] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:11.131465 29698 leader_election.cc:302] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:11.122654 30140 raft_consensus.cc:2781] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:11.132301 30140 raft_consensus.cc:482] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:11.132655 30140 raft_consensus.cc:3037] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:11.123639 30119 raft_consensus.cc:2781] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:11.139794 29852 tablet_bootstrap.cc:492] T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:11.140265 29852 ts_tablet_manager.cc:1367] T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.083s user 0.007s sys 0.010s
I20240501 20:01:11.142855 30168 raft_consensus.cc:482] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:11.142645 29852 raft_consensus.cc:348] T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:11.143498 29852 raft_consensus.cc:374] T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:11.143311 30168 raft_consensus.cc:504] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:11.143787 29852 raft_consensus.cc:729] T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:11.145292 30168 leader_election.cc:288] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:11.146009 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8cd40cb4044d18a86001ce6ddf97ab" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:11.146462 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8cd40cb4044d18a86001ce6ddf97ab" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:11.144599 29852 consensus_queue.cc:260] T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:11.146624 29734 raft_consensus.cc:2454] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 0.
I20240501 20:01:11.148247 30073 raft_consensus.cc:482] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240501 20:01:11.148283 29625 leader_election.cc:341] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:11.149506 29852 ts_tablet_manager.cc:1398] T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.009s user 0.006s sys 0.001s
I20240501 20:01:11.149654 29624 leader_election.cc:302] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:11.150416 29852 tablet_bootstrap.cc:492] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:11.148746 30073 raft_consensus.cc:504] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:11.152302 30073 leader_election.cc:288] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:11.154961 30168 raft_consensus.cc:2781] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:11.155160 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0374bb790ed345ed90452595f71383a2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:11.155364 30168 raft_consensus.cc:482] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:11.155833 30168 raft_consensus.cc:3037] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:11.156653 29852 tablet_bootstrap.cc:654] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:11.137025 30238 raft_consensus.cc:2726] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:11.153297 30119 raft_consensus.cc:686] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:11.138731 30236 consensus_queue.cc:1035] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.029s
W20240501 20:01:11.177295 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 43882.8us
I20240501 20:01:11.176203 30187 raft_consensus.cc:504] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:11.199285 29849 tablet_bootstrap.cc:492] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:11.199896 29849 ts_tablet_manager.cc:1367] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.244s user 0.021s sys 0.017s
I20240501 20:01:11.202984 29849 raft_consensus.cc:348] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:11.203657 29849 raft_consensus.cc:374] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:11.203891 29849 raft_consensus.cc:729] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:11.204447 29849 consensus_queue.cc:260] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:11.204681 30119 consensus_queue.cc:237] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:11.206573 29849 ts_tablet_manager.cc:1398] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240501 20:01:11.214596 29849 tablet_bootstrap.cc:492] T ec073fe645614fe28e7837de3c22510b P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:11.218153 30167 raft_consensus.cc:482] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:11.218773 30167 raft_consensus.cc:504] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:11.233059 29849 tablet_bootstrap.cc:654] T ec073fe645614fe28e7837de3c22510b P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:11.236297 30187 leader_election.cc:288] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:11.237712 29851 tablet_bootstrap.cc:654] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:11.209383 30140 raft_consensus.cc:504] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:11.239871 30167 leader_election.cc:288] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:11.241621 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ed27033ab1645e9ab0600ab635d9589" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:11.243213 29651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:11.244256 29651 raft_consensus.cc:3037] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:11.246855 30168 raft_consensus.cc:504] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:11.248598 30140 leader_election.cc:288] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:11.250789 29625 leader_election.cc:341] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:11.262598 30168 leader_election.cc:288] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:11.271905 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:11.272728 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8cd40cb4044d18a86001ce6ddf97ab" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:11.252457 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:11.274838 29809 raft_consensus.cc:2379] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d in current term 1: Already voted for candidate ee0d97247d394a8facfa774f15f3a67d in this term.
W20240501 20:01:11.276897 29625 leader_election.cc:341] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:11.294148 29698 leader_election.cc:302] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:11.295367 29852 tablet_bootstrap.cc:492] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:11.295966 29852 ts_tablet_manager.cc:1367] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.146s user 0.060s sys 0.031s
I20240501 20:01:11.299741 30238 raft_consensus.cc:2726] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 20:01:11.317605 30240 raft_consensus.cc:482] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:11.318117 30240 raft_consensus.cc:504] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:11.305117 30140 raft_consensus.cc:482] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:11.318966 30140 raft_consensus.cc:504] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:11.322046 29606 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240501 20:01:11.336301 30168 raft_consensus.cc:482] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:11.337517 30168 raft_consensus.cc:504] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:11.339646 30168 leader_election.cc:288] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:11.340282 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e49dd5180f04313ab33942fb3ddc147" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:11.340490 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e49dd5180f04313ab33942fb3ddc147" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:11.342067 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8787bc4d6b44649b9138d906730110a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:11.343217 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8787bc4d6b44649b9138d906730110a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:11.344525 29700 leader_election.cc:341] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
W20240501 20:01:11.346477 29700 leader_election.cc:341] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
W20240501 20:01:11.350883 29698 leader_election.cc:341] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:11.314852 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0374bb790ed345ed90452595f71383a2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:11.320009 30240 leader_election.cc:288] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:11.355561 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:11.356402 29733 raft_consensus.cc:2379] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 1: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:11.356343 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36057b3f24d44b9aac3a1378053a395f" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:11.324262 30167 raft_consensus.cc:482] T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:11.357798 30167 raft_consensus.cc:504] T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:11.358196 29651 raft_consensus.cc:2454] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
I20240501 20:01:11.359674 30167 leader_election.cc:288] T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:11.360162 29652 raft_consensus.cc:3037] T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:11.360244 29852 raft_consensus.cc:348] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:11.360971 29852 raft_consensus.cc:374] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:11.361255 29852 raft_consensus.cc:729] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:11.361862 29852 consensus_queue.cc:260] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:11.364840 29851 tablet_bootstrap.cc:492] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:11.365280 29772 leader_election.cc:302] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:11.320955 30140 leader_election.cc:288] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:11.365438 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f5d1686214485d91a0baf4b870740e" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:11.367775 30236 raft_consensus.cc:2781] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:11.368237 30236 raft_consensus.cc:686] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:11.369174 30236 consensus_queue.cc:237] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:11.365310 29851 ts_tablet_manager.cc:1367] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.271s user 0.004s sys 0.015s
I20240501 20:01:11.351481 29698 leader_election.cc:302] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:11.374883 29773 leader_election.cc:302] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
W20240501 20:01:11.380980 29698 leader_election.cc:341] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:11.381361 29698 leader_election.cc:302] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:11.385905 30073 raft_consensus.cc:2726] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:11.382505 29557 catalog_manager.cc:5484] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 0 to 1, leader changed from <none> to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 1 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:11.391039 29852 ts_tablet_manager.cc:1398] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.095s user 0.007s sys 0.000s
I20240501 20:01:11.392496 30073 raft_consensus.cc:482] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:11.392992 30073 raft_consensus.cc:504] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:11.394392 30073 leader_election.cc:288] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:11.395541 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a35893b91247a99912eba787d787cf" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:11.396266 29989 raft_consensus.cc:2726] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240501 20:01:11.397614 29772 leader_election.cc:341] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:11.402904 29849 tablet_bootstrap.cc:492] T ec073fe645614fe28e7837de3c22510b P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:11.403460 29849 ts_tablet_manager.cc:1367] T ec073fe645614fe28e7837de3c22510b P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.189s user 0.015s sys 0.009s
I20240501 20:01:11.407847 29849 raft_consensus.cc:348] T ec073fe645614fe28e7837de3c22510b P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:11.413403 29849 raft_consensus.cc:374] T ec073fe645614fe28e7837de3c22510b P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:11.413707 29849 raft_consensus.cc:729] T ec073fe645614fe28e7837de3c22510b P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:11.409837 29852 tablet_bootstrap.cc:492] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:11.409973 29804 raft_consensus.cc:3037] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:11.397771 30140 raft_consensus.cc:2726] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:11.418170 29851 raft_consensus.cc:348] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:11.418891 29851 raft_consensus.cc:374] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:11.423087 29851 raft_consensus.cc:729] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:11.423563 29851 consensus_queue.cc:260] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:11.430889 29849 consensus_queue.cc:260] T ec073fe645614fe28e7837de3c22510b P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:11.430385 29557 catalog_manager.cc:5484] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 0 to 1, leader changed from <none> to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 1 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:11.438938 29851 ts_tablet_manager.cc:1398] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.068s user 0.004s sys 0.001s
I20240501 20:01:11.439898 29851 tablet_bootstrap.cc:492] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d: Bootstrap starting.
I20240501 20:01:11.440090 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8cd40cb4044d18a86001ce6ddf97ab" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:11.440541 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ed27033ab1645e9ab0600ab635d9589" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:11.440735 29728 raft_consensus.cc:3037] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:11.442622 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36057b3f24d44b9aac3a1378053a395f" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:11.445315 29730 raft_consensus.cc:2454] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 0.
I20240501 20:01:11.447309 29849 ts_tablet_manager.cc:1398] T ec073fe645614fe28e7837de3c22510b P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.043s user 0.006s sys 0.000s
I20240501 20:01:11.448258 29849 tablet_bootstrap.cc:492] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:11.449095 29728 raft_consensus.cc:2454] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:11.450608 30240 raft_consensus.cc:482] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:11.451614 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a35893b91247a99912eba787d787cf" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:11.455476 30168 consensus_queue.cc:1035] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:11.461105 30240 raft_consensus.cc:504] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:11.463590 29773 leader_election.cc:341] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:11.464185 29773 leader_election.cc:302] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:11.464956 30073 raft_consensus.cc:2726] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:11.466845 30240 leader_election.cc:288] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:11.468158 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84fa28fb55f44b61ba1a88ed775a78ba" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:11.470149 29700 leader_election.cc:341] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:11.478992 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f5d1686214485d91a0baf4b870740e" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:11.479686 29734 raft_consensus.cc:2379] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 1: Already voted for candidate ee0d97247d394a8facfa774f15f3a67d in this term.
I20240501 20:01:11.480352 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84fa28fb55f44b61ba1a88ed775a78ba" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:11.430619 30236 consensus_queue.cc:1035] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.047s
I20240501 20:01:11.485268 29985 consensus_queue.cc:1035] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:01:11.487028 29698 leader_election.cc:341] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:11.487481 29852 tablet_bootstrap.cc:654] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:11.487833 29849 tablet_bootstrap.cc:654] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:11.487473 29698 leader_election.cc:302] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:11.489291 30240 raft_consensus.cc:2726] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:11.493697 30073 consensus_queue.cc:1035] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20240501 20:01:11.518216 29851 tablet_bootstrap.cc:654] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:11.556658 29624 leader_election.cc:302] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:11.557732 29624 leader_election.cc:302] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:11.558295 30166 raft_consensus.cc:2726] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:11.558885 30166 raft_consensus.cc:2781] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:11.559206 30166 raft_consensus.cc:482] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:11.568570 30166 raft_consensus.cc:3037] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:11.616897 29977 consensus_queue.cc:1035] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:01:11.621412 29624 leader_election.cc:302] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:11.625140 29624 leader_election.cc:302] T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:11.629103 30167 raft_consensus.cc:2781] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:11.629308 30168 raft_consensus.cc:2726] T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:11.674741 29849 tablet_bootstrap.cc:492] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:11.675424 29849 ts_tablet_manager.cc:1367] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.227s user 0.023s sys 0.000s
I20240501 20:01:11.678699 30166 raft_consensus.cc:504] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:11.682096 29851 tablet_bootstrap.cc:492] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d: No bootstrap required, opened a new log
I20240501 20:01:11.681901 29849 raft_consensus.cc:348] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:11.682850 30166 leader_election.cc:288] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:11.684645 29849 raft_consensus.cc:374] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:11.684973 29849 raft_consensus.cc:729] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:11.685974 29849 consensus_queue.cc:260] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:11.687798 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 60952us
W20240501 20:01:11.690376 29700 consensus_peers.cc:485] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d -> Peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Couldn't send request to peer ee0d97247d394a8facfa774f15f3a67d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:11.696270 29849 ts_tablet_manager.cc:1398] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.021s user 0.003s sys 0.004s
I20240501 20:01:11.688107 29851 ts_tablet_manager.cc:1367] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d: Time spent bootstrapping tablet: real 0.248s user 0.020s sys 0.012s
I20240501 20:01:11.697273 29849 tablet_bootstrap.cc:492] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:11.702898 29852 tablet_bootstrap.cc:492] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:11.703464 29852 ts_tablet_manager.cc:1367] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.294s user 0.038s sys 0.033s
W20240501 20:01:11.688707 30254 log.cc:924] Time spent T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20240501 20:01:11.688143 30167 raft_consensus.cc:686] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:11.704538 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ed27033ab1645e9ab0600ab635d9589" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
W20240501 20:01:11.717722 29625 leader_election.cc:341] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:11.725768 29852 raft_consensus.cc:348] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:11.726883 29852 raft_consensus.cc:374] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:11.727254 29852 raft_consensus.cc:729] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:11.728195 29852 consensus_queue.cc:260] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:11.741494 29852 ts_tablet_manager.cc:1398] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.038s user 0.007s sys 0.001s
I20240501 20:01:11.739805 30263 raft_consensus.cc:482] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:11.743054 29852 tablet_bootstrap.cc:492] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:11.743958 30263 raft_consensus.cc:504] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:11.752035 29652 raft_consensus.cc:3037] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:11.752666 30263 leader_election.cc:288] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:11.760797 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec073fe645614fe28e7837de3c22510b" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:11.761487 29804 raft_consensus.cc:2454] T ec073fe645614fe28e7837de3c22510b P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 0.
I20240501 20:01:11.764417 29625 leader_election.cc:302] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:11.764889 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ed27033ab1645e9ab0600ab635d9589" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:11.765486 29732 raft_consensus.cc:3037] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:11.765789 30262 raft_consensus.cc:2781] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:11.766433 30262 raft_consensus.cc:482] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:11.766832 30262 raft_consensus.cc:3037] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:11.767064 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec073fe645614fe28e7837de3c22510b" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:11.767697 29730 raft_consensus.cc:2454] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 0.
I20240501 20:01:11.770574 30263 consensus_queue.cc:1035] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:01:11.769016 30241 consensus_queue.cc:1035] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:01:11.745874 30167 consensus_queue.cc:237] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:11.777562 29849 tablet_bootstrap.cc:654] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:11.787674 29851 raft_consensus.cc:348] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:11.796733 29851 raft_consensus.cc:374] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:11.797062 29851 raft_consensus.cc:729] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Initialized, Role: FOLLOWER
I20240501 20:01:11.797755 29851 consensus_queue.cc:260] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:11.803184 30241 consensus_queue.cc:1035] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:11.803773 29977 consensus_queue.cc:1035] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:11.782218 30262 raft_consensus.cc:504] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:11.809881 29851 ts_tablet_manager.cc:1398] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d: Time spent starting tablet: real 0.113s user 0.009s sys 0.002s
I20240501 20:01:11.816218 30240 raft_consensus.cc:482] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:11.831559 30244 raft_consensus.cc:482] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:11.838325 30244 raft_consensus.cc:504] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:11.840308 30262 leader_election.cc:288] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:11.841197 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec073fe645614fe28e7837de3c22510b" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:11.841851 29807 raft_consensus.cc:3037] T ec073fe645614fe28e7837de3c22510b P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:11.849188 29852 tablet_bootstrap.cc:654] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:11.821015 30240 raft_consensus.cc:504] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:11.925593 30249 consensus_queue.cc:1035] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.074s
I20240501 20:01:11.934311 30168 raft_consensus.cc:482] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:11.934801 30168 raft_consensus.cc:504] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:11.936388 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec073fe645614fe28e7837de3c22510b" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:11.936545 29807 raft_consensus.cc:2454] T ec073fe645614fe28e7837de3c22510b P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:11.937032 29731 raft_consensus.cc:3037] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:11.937099 30168 leader_election.cc:288] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:11.924623 30236 raft_consensus.cc:482] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:11.947839 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31fff64b453a423eb0f567b3171564fb" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:11.983641 29732 raft_consensus.cc:2454] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:11.984627 29657 raft_consensus.cc:2454] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 0.
I20240501 20:01:11.970846 30244 leader_election.cc:288] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:11.966671 30236 raft_consensus.cc:504] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:12.022730 30236 leader_election.cc:288] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:12.027724 30240 leader_election.cc:288] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:12.065160 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48fd0a8e289401aa7417c42ff2c81b2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:12.065969 29734 raft_consensus.cc:2454] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
W20240501 20:01:12.072198 30270 log.cc:924] Time spent T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.101s user 0.000s sys 0.000s
I20240501 20:01:12.074174 29624 leader_election.cc:302] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:12.076182 30166 raft_consensus.cc:2781] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:12.076678 30166 raft_consensus.cc:686] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:12.077268 29624 leader_election.cc:302] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:12.077457 30166 consensus_queue.cc:237] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:12.080452 29849 tablet_bootstrap.cc:492] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:12.080843 29849 ts_tablet_manager.cc:1367] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.384s user 0.014s sys 0.009s
I20240501 20:01:12.080910 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31fff64b453a423eb0f567b3171564fb" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:12.081710 29807 raft_consensus.cc:2454] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 0.
I20240501 20:01:12.082896 29849 raft_consensus.cc:348] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:12.083508 29849 raft_consensus.cc:374] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:12.083698 29849 raft_consensus.cc:729] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:12.083761 29731 raft_consensus.cc:2454] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:12.084957 29624 leader_election.cc:302] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:12.087949 30269 raft_consensus.cc:2781] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:12.088371 30269 raft_consensus.cc:482] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:12.088706 30269 raft_consensus.cc:3037] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:12.096644 30265 raft_consensus.cc:2781] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:12.097205 30265 raft_consensus.cc:686] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:12.097721 29849 consensus_queue.cc:260] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:12.098014 30265 consensus_queue.cc:237] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:12.100080 29849 ts_tablet_manager.cc:1398] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.019s user 0.005s sys 0.001s
I20240501 20:01:12.101491 29849 tablet_bootstrap.cc:492] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
W20240501 20:01:12.103169 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 131890us
I20240501 20:01:12.110723 29852 tablet_bootstrap.cc:492] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:12.111198 29852 ts_tablet_manager.cc:1367] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.368s user 0.011s sys 0.010s
I20240501 20:01:12.122632 29554 catalog_manager.cc:5484] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 1 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:12.135917 29849 tablet_bootstrap.cc:654] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:12.139035 29700 leader_election.cc:302] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:12.141412 30140 raft_consensus.cc:2781] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:12.141826 30140 raft_consensus.cc:482] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:12.142724 30140 raft_consensus.cc:3037] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:12.149176 30284 raft_consensus.cc:482] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:12.149691 30284 raft_consensus.cc:504] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:12.151197 30284 leader_election.cc:288] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:12.159377 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a35893b91247a99912eba787d787cf" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:12.160269 29808 raft_consensus.cc:2454] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 0.
I20240501 20:01:12.165889 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48fd0a8e289401aa7417c42ff2c81b2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:12.158756 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31fff64b453a423eb0f567b3171564fb" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
W20240501 20:01:12.218152 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 56667.2us
I20240501 20:01:12.142938 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48fd0a8e289401aa7417c42ff2c81b2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:12.228155 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 63387.8us
I20240501 20:01:12.145951 30269 raft_consensus.cc:504] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:12.244851 30269 leader_election.cc:288] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:12.192963 29654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31fff64b453a423eb0f567b3171564fb" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:12.257400 29660 raft_consensus.cc:2354] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d for earlier term 2. Current term is 3.
I20240501 20:01:12.236001 29804 raft_consensus.cc:2454] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 2.
I20240501 20:01:12.252983 30274 raft_consensus.cc:482] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:12.349406 29654 raft_consensus.cc:2454] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 0.
I20240501 20:01:12.374013 29852 raft_consensus.cc:348] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:12.374806 29852 raft_consensus.cc:374] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:12.375097 29852 raft_consensus.cc:729] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:12.375655 29852 consensus_queue.cc:260] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:12.388028 29852 ts_tablet_manager.cc:1398] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.277s user 0.008s sys 0.001s
I20240501 20:01:12.391448 29700 leader_election.cc:302] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:12.358006 30274 raft_consensus.cc:504] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:12.405978 29849 tablet_bootstrap.cc:492] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:12.407344 29849 ts_tablet_manager.cc:1367] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.306s user 0.009s sys 0.011s
I20240501 20:01:12.400540 30238 raft_consensus.cc:2781] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:12.435652 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a35893b91247a99912eba787d787cf" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:12.442184 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48fd0a8e289401aa7417c42ff2c81b2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:12.443033 29805 raft_consensus.cc:3037] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:12.420406 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48fd0a8e289401aa7417c42ff2c81b2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:12.472923 29849 raft_consensus.cc:348] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:12.504083 29849 raft_consensus.cc:374] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:12.504406 29849 raft_consensus.cc:729] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:12.505028 29849 consensus_queue.cc:260] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:12.501227 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 105434us
I20240501 20:01:12.510500 29700 leader_election.cc:302] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:12.431617 30238 raft_consensus.cc:482] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240501 20:01:12.523041 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 78317.4us
W20240501 20:01:12.524760 29698 leader_election.cc:341] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:12.475389 30263 raft_consensus.cc:482] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:12.539475 29852 tablet_bootstrap.cc:492] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:12.513703 29729 raft_consensus.cc:3037] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 3
I20240501 20:01:12.528560 30238 raft_consensus.cc:3037] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
W20240501 20:01:12.552312 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 149715us
I20240501 20:01:12.557540 29849 ts_tablet_manager.cc:1398] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.149s user 0.010s sys 0.001s
W20240501 20:01:12.564558 29772 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 137013us
W20240501 20:01:12.572784 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 47006.9us
I20240501 20:01:12.538365 30263 raft_consensus.cc:504] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:12.577580 29772 leader_election.cc:302] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:12.586799 29852 tablet_bootstrap.cc:654] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:12.587078 30295 raft_consensus.cc:2781] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:12.587628 30295 raft_consensus.cc:482] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:12.588086 30295 raft_consensus.cc:3037] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:12.572762 30274 leader_election.cc:288] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:12.606393 30263 leader_election.cc:288] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:12.626574 29809 raft_consensus.cc:3037] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:12.621798 30140 raft_consensus.cc:504] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:12.673070 30140 leader_election.cc:288] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:12.687036 29800 raft_consensus.cc:3037] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:12.696863 29728 raft_consensus.cc:2379] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 1: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:12.735024 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48fd0a8e289401aa7417c42ff2c81b2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:12.738817 29849 tablet_bootstrap.cc:492] T a8787bc4d6b44649b9138d906730110a P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:12.760085 29729 raft_consensus.cc:2454] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 3.
I20240501 20:01:12.760711 29989 raft_consensus.cc:2741] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20240501 20:01:12.742734 29805 raft_consensus.cc:2454] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 3.
I20240501 20:01:12.765285 29849 tablet_bootstrap.cc:654] T a8787bc4d6b44649b9138d906730110a P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
W20240501 20:01:12.789000 29772 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 210314us
W20240501 20:01:12.788385 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 74479us
I20240501 20:01:12.710841 30238 raft_consensus.cc:504] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:12.828259 30275 consensus_queue.cc:1035] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
W20240501 20:01:12.828662 29606 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240501 20:01:12.774629 30295 raft_consensus.cc:504] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:12.830768 29698 leader_election.cc:398] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Vote denied by peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959) with higher term. Message: Invalid argument: T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d for earlier term 2. Current term is 3.
I20240501 20:01:12.840653 29625 leader_election.cc:302] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:12.849069 30168 raft_consensus.cc:2781] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:01:12.849527 30168 raft_consensus.cc:686] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:12.850315 30168 consensus_queue.cc:237] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:12.843021 30295 leader_election.cc:288] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:12.847312 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31fff64b453a423eb0f567b3171564fb" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:12.862509 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31fff64b453a423eb0f567b3171564fb" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:12.863561 29734 raft_consensus.cc:2379] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 1: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:12.865020 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8edcd903224d48f889b46e1b5318e0ec" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:12.858769 29652 raft_consensus.cc:3037] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
W20240501 20:01:12.898945 29625 consensus_peers.cc:485] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f -> Peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Couldn't send request to peer ee0d97247d394a8facfa774f15f3a67d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:12.857404 30263 consensus_queue.cc:1035] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.030s
I20240501 20:01:12.886425 29654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31fff64b453a423eb0f567b3171564fb" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:12.933840 29852 tablet_bootstrap.cc:492] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:12.934285 29852 ts_tablet_manager.cc:1367] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.395s user 0.046s sys 0.036s
W20240501 20:01:12.938519 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 48580.5us
W20240501 20:01:12.943260 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 56293.4us
I20240501 20:01:12.963030 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8787bc4d6b44649b9138d906730110a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:12.966706 29734 raft_consensus.cc:2454] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 0.
I20240501 20:01:12.968195 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a35893b91247a99912eba787d787cf" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:12.976269 30265 raft_consensus.cc:482] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:12.979736 30265 raft_consensus.cc:504] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:12.982195 30265 leader_election.cc:288] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:12.984818 30312 log.cc:924] Time spent T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20240501 20:01:12.985320 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:12.937916 30269 consensus_queue.cc:1035] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.041s
I20240501 20:01:12.967832 30238 leader_election.cc:288] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:13.024761 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 58735.3us
W20240501 20:01:13.026551 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 56199us
I20240501 20:01:13.027458 29849 tablet_bootstrap.cc:492] T a8787bc4d6b44649b9138d906730110a P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:13.028048 29849 ts_tablet_manager.cc:1367] T a8787bc4d6b44649b9138d906730110a P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.289s user 0.020s sys 0.039s
I20240501 20:01:13.028785 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8787bc4d6b44649b9138d906730110a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:13.054912 29772 leader_election.cc:302] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:13.056771 29852 raft_consensus.cc:348] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:13.057663 29852 raft_consensus.cc:374] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:13.057925 29852 raft_consensus.cc:729] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:13.057853 29624 leader_election.cc:302] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:13.058619 29852 consensus_queue.cc:260] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:13.063758 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:13.068125 29852 ts_tablet_manager.cc:1398] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.134s user 0.006s sys 0.000s
W20240501 20:01:13.067117 29625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 61094.6us
W20240501 20:01:13.072829 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 92507.4us
I20240501 20:01:13.072760 29624 leader_election.cc:302] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:13.068338 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8edcd903224d48f889b46e1b5318e0ec" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:13.081461 30167 raft_consensus.cc:2726] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 20:01:13.129164 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 55031.9us
I20240501 20:01:13.086238 30295 raft_consensus.cc:2726] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 20:01:13.101191 30261 raft_consensus.cc:2781] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:13.148283 29773 leader_election.cc:302] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:13.127741 30265 consensus_queue.cc:1035] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20240501 20:01:13.140666 30261 raft_consensus.cc:482] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:13.159129 29849 raft_consensus.cc:348] T a8787bc4d6b44649b9138d906730110a P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:13.160192 29849 raft_consensus.cc:374] T a8787bc4d6b44649b9138d906730110a P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:13.160558 29849 raft_consensus.cc:729] T a8787bc4d6b44649b9138d906730110a P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:13.161327 29849 consensus_queue.cc:260] T a8787bc4d6b44649b9138d906730110a P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:13.148638 29652 raft_consensus.cc:2454] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
I20240501 20:01:13.182899 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31fff64b453a423eb0f567b3171564fb" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:13.184108 29807 raft_consensus.cc:2379] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d in current term 1: Already voted for candidate ee0d97247d394a8facfa774f15f3a67d in this term.
I20240501 20:01:13.187794 29659 raft_consensus.cc:3037] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:13.192715 29852 tablet_bootstrap.cc:492] T 84fa28fb55f44b61ba1a88ed775a78ba P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:13.166285 30073 raft_consensus.cc:2726] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:13.167349 30308 raft_consensus.cc:482] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:13.195191 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a35893b91247a99912eba787d787cf" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:13.195817 29809 raft_consensus.cc:3037] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:13.195719 30308 raft_consensus.cc:504] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:13.197836 30308 leader_election.cc:288] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:13.202020 29809 raft_consensus.cc:2454] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
I20240501 20:01:13.164949 30261 raft_consensus.cc:3037] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
W20240501 20:01:13.208889 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 58739us
I20240501 20:01:13.212770 29700 leader_election.cc:302] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:13.215225 29852 tablet_bootstrap.cc:654] T 84fa28fb55f44b61ba1a88ed775a78ba P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:13.218746 30319 consensus_queue.cc:1035] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20240501 20:01:13.204981 29659 raft_consensus.cc:2454] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
I20240501 20:01:13.239540 30279 raft_consensus.cc:2781] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election won for term 1
W20240501 20:01:13.286460 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 79085us
I20240501 20:01:13.289170 29698 leader_election.cc:302] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:13.247601 30279 raft_consensus.cc:686] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:13.279685 29806 raft_consensus.cc:2454] T a8787bc4d6b44649b9138d906730110a P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 0.
I20240501 20:01:13.306460 30282 raft_consensus.cc:2781] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:13.309494 29849 ts_tablet_manager.cc:1398] T a8787bc4d6b44649b9138d906730110a P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.281s user 0.006s sys 0.052s
I20240501 20:01:13.295321 30279 consensus_queue.cc:237] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:13.335395 30282 raft_consensus.cc:686] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:13.355911 30303 raft_consensus.cc:482] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:13.356540 30303 raft_consensus.cc:504] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:13.345317 30282 consensus_queue.cc:237] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:13.372696 30303 leader_election.cc:288] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:13.373901 30261 raft_consensus.cc:504] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:13.376209 30261 leader_election.cc:288] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:13.378953 29852 tablet_bootstrap.cc:492] T 84fa28fb55f44b61ba1a88ed775a78ba P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:13.379623 29852 ts_tablet_manager.cc:1367] T 84fa28fb55f44b61ba1a88ed775a78ba P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.187s user 0.019s sys 0.009s
I20240501 20:01:13.422770 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8cd40cb4044d18a86001ce6ddf97ab" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:13.454573 30317 raft_consensus.cc:482] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:13.465888 29849 tablet_bootstrap.cc:492] T 7ed27033ab1645e9ab0600ab635d9589 P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:13.481218 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8787bc4d6b44649b9138d906730110a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:13.481515 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84fa28fb55f44b61ba1a88ed775a78ba" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:13.482110 29803 raft_consensus.cc:3037] T a8787bc4d6b44649b9138d906730110a P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:13.478197 30317 raft_consensus.cc:504] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:13.506310 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8cd40cb4044d18a86001ce6ddf97ab" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:13.516930 30317 leader_election.cc:288] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:13.540102 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84fa28fb55f44b61ba1a88ed775a78ba" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:13.537015 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8787bc4d6b44649b9138d906730110a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:13.541607 29733 raft_consensus.cc:3037] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:13.551919 29733 raft_consensus.cc:2454] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:13.569967 29803 raft_consensus.cc:2454] T a8787bc4d6b44649b9138d906730110a P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
W20240501 20:01:13.566063 29700 leader_election.cc:341] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:13.613225 29556 catalog_manager.cc:5484] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 1 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } }
I20240501 20:01:13.619117 29624 leader_election.cc:302] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:13.621695 29852 raft_consensus.cc:348] T 84fa28fb55f44b61ba1a88ed775a78ba P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:13.622648 29852 raft_consensus.cc:374] T 84fa28fb55f44b61ba1a88ed775a78ba P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:13.622982 29852 raft_consensus.cc:729] T 84fa28fb55f44b61ba1a88ed775a78ba P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
W20240501 20:01:13.624176 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 44689.4us
W20240501 20:01:13.625948 29698 leader_election.cc:341] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
W20240501 20:01:13.608748 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 44606.6us
I20240501 20:01:13.626350 29698 leader_election.cc:302] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:13.623708 29852 consensus_queue.cc:260] T 84fa28fb55f44b61ba1a88ed775a78ba P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:13.631377 30243 raft_consensus.cc:2726] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:13.631749 29852 ts_tablet_manager.cc:1398] T 84fa28fb55f44b61ba1a88ed775a78ba P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.252s user 0.003s sys 0.004s
I20240501 20:01:13.626367 29732 raft_consensus.cc:1261] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240501 20:01:13.634666 29852 tablet_bootstrap.cc:492] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f: Bootstrap starting.
I20240501 20:01:13.648305 29801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e49dd5180f04313ab33942fb3ddc147" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:13.650254 29700 leader_election.cc:341] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:13.652004 30320 consensus_queue.cc:1035] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 20:01:13.624444 30265 raft_consensus.cc:2781] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:13.654175 30265 raft_consensus.cc:686] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:13.653616 29801 raft_consensus.cc:1261] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240501 20:01:13.665369 29773 leader_election.cc:302] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:13.669585 29654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e49dd5180f04313ab33942fb3ddc147" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:13.670117 30187 raft_consensus.cc:2726] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:13.661899 30265 consensus_queue.cc:237] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:13.692514 29849 tablet_bootstrap.cc:654] T 7ed27033ab1645e9ab0600ab635d9589 P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:13.706140 30166 consensus_queue.cc:1035] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20240501 20:01:13.736101 29852 tablet_bootstrap.cc:654] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:13.752828 29852 tablet_bootstrap.cc:492] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f: No bootstrap required, opened a new log
I20240501 20:01:13.753410 29852 ts_tablet_manager.cc:1367] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f: Time spent bootstrapping tablet: real 0.119s user 0.011s sys 0.003s
I20240501 20:01:13.756099 29852 raft_consensus.cc:348] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:13.756855 29852 raft_consensus.cc:374] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:13.757079 29849 tablet_bootstrap.cc:492] T 7ed27033ab1645e9ab0600ab635d9589 P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:13.757174 29852 raft_consensus.cc:729] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Initialized, Role: FOLLOWER
I20240501 20:01:13.757596 29849 ts_tablet_manager.cc:1367] T 7ed27033ab1645e9ab0600ab635d9589 P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.293s user 0.011s sys 0.004s
I20240501 20:01:13.758138 29852 consensus_queue.cc:260] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:13.760176 29849 raft_consensus.cc:348] T 7ed27033ab1645e9ab0600ab635d9589 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:13.760643 29852 ts_tablet_manager.cc:1398] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240501 20:01:13.760926 29849 raft_consensus.cc:374] T 7ed27033ab1645e9ab0600ab635d9589 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:13.761224 29849 raft_consensus.cc:729] T 7ed27033ab1645e9ab0600ab635d9589 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:13.761868 29849 consensus_queue.cc:260] T 7ed27033ab1645e9ab0600ab635d9589 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:13.763475 29849 ts_tablet_manager.cc:1398] T 7ed27033ab1645e9ab0600ab635d9589 P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
W20240501 20:01:13.764276 29698 leader_election.cc:341] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:01:13.764753 29698 leader_election.cc:302] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:13.765527 30317 raft_consensus.cc:2726] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:13.783651 29849 tablet_bootstrap.cc:492] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:13.795233 29849 tablet_bootstrap.cc:654] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:13.794593 29557 catalog_manager.cc:5484] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 1 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:13.798993 29557 catalog_manager.cc:5484] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 1 to 3, leader changed from ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 3 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:13.827725 30317 consensus_queue.cc:1035] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:13.830536 30244 consensus_queue.cc:1035] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:01:13.843916 29985 raft_consensus.cc:482] T 7ed27033ab1645e9ab0600ab635d9589 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:13.844481 29985 raft_consensus.cc:504] T 7ed27033ab1645e9ab0600ab635d9589 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:13.846346 29985 leader_election.cc:288] T 7ed27033ab1645e9ab0600ab635d9589 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:13.847182 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ed27033ab1645e9ab0600ab635d9589" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:13.854862 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ed27033ab1645e9ab0600ab635d9589" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:13.857110 29558 catalog_manager.cc:5484] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 0 to 1, leader changed from <none> to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 1 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:13.862547 29773 leader_election.cc:302] T 7ed27033ab1645e9ab0600ab635d9589 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:13.863799 29985 raft_consensus.cc:2726] T 7ed27033ab1645e9ab0600ab635d9589 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:01:13.858927 29558 catalog_manager.cc:5484] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 0 to 1, leader changed from <none> to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 1 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:13.865458 29849 tablet_bootstrap.cc:492] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:13.865906 29849 ts_tablet_manager.cc:1367] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.098s user 0.020s sys 0.006s
I20240501 20:01:13.891324 29849 raft_consensus.cc:348] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:13.896483 29849 raft_consensus.cc:374] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:13.896823 29849 raft_consensus.cc:729] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:13.897502 29849 consensus_queue.cc:260] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:13.902042 29849 ts_tablet_manager.cc:1398] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.036s user 0.007s sys 0.000s
I20240501 20:01:13.903096 29849 tablet_bootstrap.cc:492] T 84fa28fb55f44b61ba1a88ed775a78ba P ee0d97247d394a8facfa774f15f3a67d: Bootstrap starting.
I20240501 20:01:13.917783 29849 tablet_bootstrap.cc:654] T 84fa28fb55f44b61ba1a88ed775a78ba P ee0d97247d394a8facfa774f15f3a67d: Neither blocks nor log segments found. Creating new log.
I20240501 20:01:13.928293 30243 consensus_queue.cc:1035] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:01:13.965042 29606 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240501 20:01:13.974400 29556 catalog_manager.cc:5484] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 1 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:13.975148 30243 consensus_queue.cc:1035] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:13.978754 29849 tablet_bootstrap.cc:492] T 84fa28fb55f44b61ba1a88ed775a78ba P ee0d97247d394a8facfa774f15f3a67d: No bootstrap required, opened a new log
I20240501 20:01:13.979228 29849 ts_tablet_manager.cc:1367] T 84fa28fb55f44b61ba1a88ed775a78ba P ee0d97247d394a8facfa774f15f3a67d: Time spent bootstrapping tablet: real 0.076s user 0.012s sys 0.009s
I20240501 20:01:13.988853 29849 raft_consensus.cc:348] T 84fa28fb55f44b61ba1a88ed775a78ba P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:13.989567 29849 raft_consensus.cc:374] T 84fa28fb55f44b61ba1a88ed775a78ba P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:01:13.989805 29849 raft_consensus.cc:729] T 84fa28fb55f44b61ba1a88ed775a78ba P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Initialized, Role: FOLLOWER
I20240501 20:01:14.002104 29849 consensus_queue.cc:260] T 84fa28fb55f44b61ba1a88ed775a78ba P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:14.009038 29849 ts_tablet_manager.cc:1398] T 84fa28fb55f44b61ba1a88ed775a78ba P ee0d97247d394a8facfa774f15f3a67d: Time spent starting tablet: real 0.030s user 0.006s sys 0.001s
I20240501 20:01:14.104866 30263 raft_consensus.cc:482] T 84fa28fb55f44b61ba1a88ed775a78ba P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:14.105473 30263 raft_consensus.cc:504] T 84fa28fb55f44b61ba1a88ed775a78ba P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:14.108528 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84fa28fb55f44b61ba1a88ed775a78ba" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:14.109246 29801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84fa28fb55f44b61ba1a88ed775a78ba" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:14.109935 29726 raft_consensus.cc:2454] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 0.
I20240501 20:01:14.110069 29801 raft_consensus.cc:2454] T 84fa28fb55f44b61ba1a88ed775a78ba P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 0.
I20240501 20:01:14.111516 29625 leader_election.cc:302] T 84fa28fb55f44b61ba1a88ed775a78ba P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:14.112519 30261 raft_consensus.cc:2781] T 84fa28fb55f44b61ba1a88ed775a78ba P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:14.112896 30261 raft_consensus.cc:482] T 84fa28fb55f44b61ba1a88ed775a78ba P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:14.113215 30261 raft_consensus.cc:3037] T 84fa28fb55f44b61ba1a88ed775a78ba P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:14.120203 30263 leader_election.cc:288] T 84fa28fb55f44b61ba1a88ed775a78ba P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:14.124418 30261 raft_consensus.cc:504] T 84fa28fb55f44b61ba1a88ed775a78ba P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:14.130733 30261 leader_election.cc:288] T 84fa28fb55f44b61ba1a88ed775a78ba P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:14.132362 29801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84fa28fb55f44b61ba1a88ed775a78ba" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:14.133091 29801 raft_consensus.cc:3037] T 84fa28fb55f44b61ba1a88ed775a78ba P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:14.134150 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84fa28fb55f44b61ba1a88ed775a78ba" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:14.134840 29726 raft_consensus.cc:3037] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:14.151227 29800 raft_consensus.cc:3037] T 7ed27033ab1645e9ab0600ab635d9589 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:14.151964 29726 raft_consensus.cc:2454] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:14.153111 29624 leader_election.cc:302] T 84fa28fb55f44b61ba1a88ed775a78ba P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:14.155997 29801 raft_consensus.cc:2454] T 84fa28fb55f44b61ba1a88ed775a78ba P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:14.157575 30261 raft_consensus.cc:2781] T 84fa28fb55f44b61ba1a88ed775a78ba P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:14.159967 30261 raft_consensus.cc:686] T 84fa28fb55f44b61ba1a88ed775a78ba P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:14.161875 30261 consensus_queue.cc:237] T 84fa28fb55f44b61ba1a88ed775a78ba P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:14.176385 30302 consensus_queue.cc:1035] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:01:14.195385 30187 raft_consensus.cc:482] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:14.196491 30187 raft_consensus.cc:504] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:14.198544 30187 leader_election.cc:288] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:14.199002 30302 consensus_queue.cc:1035] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:14.201761 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e49dd5180f04313ab33942fb3ddc147" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:14.202394 29726 raft_consensus.cc:2454] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 0.
I20240501 20:01:14.203438 29773 leader_election.cc:302] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:14.204780 30187 raft_consensus.cc:2781] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:01:14.205117 30187 raft_consensus.cc:482] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:14.205404 30187 raft_consensus.cc:3037] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:14.211933 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e49dd5180f04313ab33942fb3ddc147" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:14.212675 29652 raft_consensus.cc:2454] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 0.
I20240501 20:01:14.219293 29558 catalog_manager.cc:5484] T 84fa28fb55f44b61ba1a88ed775a78ba P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 1 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:14.231154 30187 raft_consensus.cc:504] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:14.233757 30187 leader_election.cc:288] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:14.234938 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e49dd5180f04313ab33942fb3ddc147" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:14.236665 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e49dd5180f04313ab33942fb3ddc147" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:14.237372 29726 raft_consensus.cc:3037] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:14.237674 29653 raft_consensus.cc:3037] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:01:14.244390 29726 raft_consensus.cc:2454] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
I20240501 20:01:14.245539 29773 leader_election.cc:302] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:14.246716 30187 raft_consensus.cc:2781] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:01:14.247365 30187 raft_consensus.cc:686] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:14.248180 30187 consensus_queue.cc:237] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:14.251322 29653 raft_consensus.cc:2454] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
I20240501 20:01:14.293942 29558 catalog_manager.cc:5484] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 0 to 1, leader changed from <none> to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 1 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:14.732089 30261 consensus_queue.cc:1035] T 84fa28fb55f44b61ba1a88ed775a78ba P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:01:14.753059 30362 consensus_queue.cc:1035] T 84fa28fb55f44b61ba1a88ed775a78ba P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:01:14.777559 30324 consensus_queue.cc:1035] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:01:14.807030 30187 consensus_queue.cc:1035] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:01:14.849092 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 48437.5us
I20240501 20:01:15.107370 28519 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:01:15.134460 30375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:15.134697 30374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:01:15.140977 30377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:01:15.142136 28519 server_base.cc:1034] running on GCE node
I20240501 20:01:15.143474 28519 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 20:01:15.143865 28519 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 20:01:15.144127 28519 hybrid_clock.cc:648] HybridClock initialized: now 1714593675144109 us; error 0 us; skew 500 ppm
I20240501 20:01:15.144881 28519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:01:15.148396 28519 webserver.cc:460] Webserver started at http://127.27.217.196:46497/ using document root <none> and password file <none>
I20240501 20:01:15.148968 28519 fs_manager.cc:359] Metadata directory not provided
I20240501 20:01:15.149187 28519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:01:15.149500 28519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:01:15.151062 28519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714593602638719-28519-0/minicluster-data/ts-3-root/instance:
uuid: "888d99522e4844118f8f1cf26f268c8c"
format_stamp: "Formatted at 2024-05-01 20:01:15 on dist-test-slave-92m0"
I20240501 20:01:15.156210 28519 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20240501 20:01:15.162032 30382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:15.162950 28519 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20240501 20:01:15.163239 28519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714593602638719-28519-0/minicluster-data/ts-3-root
uuid: "888d99522e4844118f8f1cf26f268c8c"
format_stamp: "Formatted at 2024-05-01 20:01:15 on dist-test-slave-92m0"
I20240501 20:01:15.163599 28519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714593602638719-28519-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714593602638719-28519-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714593602638719-28519-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:01:15.190217 28519 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:01:15.192165 28519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:01:15.194156 28519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:01:15.197505 28519 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:01:15.197793 28519 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:15.198119 28519 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:01:15.198367 28519 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:01:15.360713 28519 rpc_server.cc:307] RPC server started. Bound to: 127.27.217.196:37227
I20240501 20:01:15.360858 30445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.217.196:37227 every 8 connection(s)
I20240501 20:01:15.373878 28519 auto_leader_rebalancer-test.cc:405] add a tserver: 888d99522e4844118f8f1cf26f268c8c
I20240501 20:01:15.392160 30446 heartbeater.cc:347] Connected to a master server at 127.27.217.254:42703
I20240501 20:01:15.392643 30446 heartbeater.cc:464] Registering TS with master...
I20240501 20:01:15.395277 30446 heartbeater.cc:511] Master 127.27.217.254:42703 requested a full tablet report, sending...
I20240501 20:01:15.398744 29558 ts_manager.cc:194] Registered new tserver with Master: 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:15.400539 29558 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33356
I20240501 20:01:16.124377 29800 consensus_queue.cc:237] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:16.134693 29732 raft_consensus.cc:1261] T add9d7579351483fb268f89a44e95d8a P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:01:16.136627 30372 consensus_queue.cc:1035] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:16.141737 29653 raft_consensus.cc:1261] T add9d7579351483fb268f89a44e95d8a P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:01:16.142928 30372 consensus_queue.cc:1035] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 20:01:16.150177 29772 consensus_peers.cc:485] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d -> Peer 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227): Couldn't send request to peer 888d99522e4844118f8f1cf26f268c8c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: add9d7579351483fb268f89a44e95d8a. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:16.153744 30370 raft_consensus.cc:2932] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
I20240501 20:01:16.156630 29733 raft_consensus.cc:2932] T add9d7579351483fb268f89a44e95d8a P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
I20240501 20:01:16.167173 29653 raft_consensus.cc:2932] T add9d7579351483fb268f89a44e95d8a P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
I20240501 20:01:16.177995 29558 catalog_manager.cc:5484] T add9d7579351483fb268f89a44e95d8a P fdfeac4190e045f290e003b2298d988d reported cstate change: config changed from index -1 to 2, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New cstate: current_term: 1 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
I20240501 20:01:16.216953 29801 consensus_queue.cc:237] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
W20240501 20:01:16.226449 29772 consensus_peers.cc:485] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d -> Peer 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227): Couldn't send request to peer 888d99522e4844118f8f1cf26f268c8c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9046e38fc57f4364aa3d539e1fc01928. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:16.227440 29733 raft_consensus.cc:1261] T 9046e38fc57f4364aa3d539e1fc01928 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:01:16.228977 30370 consensus_queue.cc:1035] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:16.229501 29653 raft_consensus.cc:1261] T 9046e38fc57f4364aa3d539e1fc01928 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:01:16.231738 30370 consensus_queue.cc:1035] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:16.250622 30370 raft_consensus.cc:2932] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
I20240501 20:01:16.252190 29733 raft_consensus.cc:2932] T 9046e38fc57f4364aa3d539e1fc01928 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
I20240501 20:01:16.267601 29653 raft_consensus.cc:2932] T 9046e38fc57f4364aa3d539e1fc01928 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
I20240501 20:01:16.295778 29652 consensus_queue.cc:237] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:16.311115 29558 catalog_manager.cc:5484] T 9046e38fc57f4364aa3d539e1fc01928 P fdfeac4190e045f290e003b2298d988d reported cstate change: config changed from index -1 to 2, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New cstate: current_term: 1 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
I20240501 20:01:16.332652 29733 raft_consensus.cc:1261] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 20:01:16.333726 29801 raft_consensus.cc:1261] T 8bd5e83faaff4e2da4e4cc22ba410d33 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 20:01:16.334228 30302 consensus_queue.cc:1035] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:16.335171 30443 consensus_queue.cc:1035] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:16.348418 29733 raft_consensus.cc:2932] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
I20240501 20:01:16.354570 30443 raft_consensus.cc:2932] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
I20240501 20:01:16.377995 29558 catalog_manager.cc:5484] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d reported cstate change: config changed from index -1 to 3, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New cstate: current_term: 3 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
I20240501 20:01:16.403139 30446 heartbeater.cc:503] Master 127.27.217.254:42703 was elected leader, sending a full tablet report...
I20240501 20:01:16.396831 29801 raft_consensus.cc:2932] T 8bd5e83faaff4e2da4e4cc22ba410d33 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
W20240501 20:01:16.414505 29623 consensus_peers.cc:485] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f -> Peer 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227): Couldn't send request to peer 888d99522e4844118f8f1cf26f268c8c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8bd5e83faaff4e2da4e4cc22ba410d33. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:16.439440 29733 consensus_queue.cc:237] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:16.460299 29801 raft_consensus.cc:1261] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:01:16.465548 30450 consensus_queue.cc:1035] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:16.468915 29652 raft_consensus.cc:1261] T cb05df43bdfa479b902b1da297fb5974 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:01:16.490540 30450 raft_consensus.cc:2932] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
I20240501 20:01:16.493171 30317 consensus_queue.cc:1035] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:16.496662 29801 raft_consensus.cc:2932] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
W20240501 20:01:16.511063 29698 consensus_peers.cc:485] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d -> Peer 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227): Couldn't send request to peer 888d99522e4844118f8f1cf26f268c8c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cb05df43bdfa479b902b1da297fb5974. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:16.510478 29554 catalog_manager.cc:5484] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d reported cstate change: config changed from index -1 to 2, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New cstate: current_term: 1 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:16.550892 30479 ts_tablet_manager.cc:902] T add9d7579351483fb268f89a44e95d8a P 888d99522e4844118f8f1cf26f268c8c: Initiating tablet copy from peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:16.599148 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 62240.7us
I20240501 20:01:16.617655 29652 raft_consensus.cc:2932] T cb05df43bdfa479b902b1da297fb5974 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
I20240501 20:01:16.638499 30479 tablet_copy_client.cc:308] T add9d7579351483fb268f89a44e95d8a P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Beginning tablet copy session from remote peer at address 127.27.217.195:35861
I20240501 20:01:16.663441 29819 tablet_copy_service.cc:140] P ee0d97247d394a8facfa774f15f3a67d: Received BeginTabletCopySession request for tablet add9d7579351483fb268f89a44e95d8a from peer 888d99522e4844118f8f1cf26f268c8c ({username='slave'} at 127.0.0.1:49474)
I20240501 20:01:16.664177 29819 tablet_copy_service.cc:161] P ee0d97247d394a8facfa774f15f3a67d: Beginning new tablet copy session on tablet add9d7579351483fb268f89a44e95d8a from peer 888d99522e4844118f8f1cf26f268c8c at {username='slave'} at 127.0.0.1:49474: session id = 888d99522e4844118f8f1cf26f268c8c-add9d7579351483fb268f89a44e95d8a
I20240501 20:01:16.678076 29819 tablet_copy_source_session.cc:205] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:01:16.682768 30479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet add9d7579351483fb268f89a44e95d8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:16.709055 30479 tablet_copy_client.cc:783] T add9d7579351483fb268f89a44e95d8a P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Starting download of 0 data blocks...
I20240501 20:01:16.710587 30479 tablet_copy_client.cc:647] T add9d7579351483fb268f89a44e95d8a P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Starting download of 1 WAL segments...
I20240501 20:01:16.757637 30479 tablet_copy_client.cc:523] T add9d7579351483fb268f89a44e95d8a P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240501 20:01:16.794564 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 76738us
I20240501 20:01:16.796589 30479 tablet_bootstrap.cc:492] T add9d7579351483fb268f89a44e95d8a P 888d99522e4844118f8f1cf26f268c8c: Bootstrap starting.
I20240501 20:01:16.845191 30490 ts_tablet_manager.cc:902] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c: Initiating tablet copy from peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:16.845726 30479 tablet_bootstrap.cc:492] T add9d7579351483fb268f89a44e95d8a P 888d99522e4844118f8f1cf26f268c8c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:01:16.846788 30479 tablet_bootstrap.cc:492] T add9d7579351483fb268f89a44e95d8a P 888d99522e4844118f8f1cf26f268c8c: Bootstrap complete.
I20240501 20:01:16.847478 30479 ts_tablet_manager.cc:1367] T add9d7579351483fb268f89a44e95d8a P 888d99522e4844118f8f1cf26f268c8c: Time spent bootstrapping tablet: real 0.051s user 0.015s sys 0.017s
I20240501 20:01:16.850441 30479 raft_consensus.cc:348] T add9d7579351483fb268f89a44e95d8a P 888d99522e4844118f8f1cf26f268c8c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:16.851398 30479 raft_consensus.cc:729] T add9d7579351483fb268f89a44e95d8a P 888d99522e4844118f8f1cf26f268c8c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 888d99522e4844118f8f1cf26f268c8c, State: Initialized, Role: LEARNER
I20240501 20:01:16.852092 30479 consensus_queue.cc:260] T add9d7579351483fb268f89a44e95d8a P 888d99522e4844118f8f1cf26f268c8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:16.876703 30479 ts_tablet_manager.cc:1398] T add9d7579351483fb268f89a44e95d8a P 888d99522e4844118f8f1cf26f268c8c: Time spent starting tablet: real 0.029s user 0.011s sys 0.007s
I20240501 20:01:16.881057 29819 tablet_copy_service.cc:342] P ee0d97247d394a8facfa774f15f3a67d: Request end of tablet copy session 888d99522e4844118f8f1cf26f268c8c-add9d7579351483fb268f89a44e95d8a received from {username='slave'} at 127.0.0.1:49474
I20240501 20:01:16.881688 29819 tablet_copy_service.cc:433] P ee0d97247d394a8facfa774f15f3a67d: ending tablet copy session 888d99522e4844118f8f1cf26f268c8c-add9d7579351483fb268f89a44e95d8a on tablet add9d7579351483fb268f89a44e95d8a with peer 888d99522e4844118f8f1cf26f268c8c
I20240501 20:01:16.896929 30494 ts_tablet_manager.cc:902] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c: Initiating tablet copy from peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:16.916500 30490 tablet_copy_client.cc:308] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Beginning tablet copy session from remote peer at address 127.27.217.195:35861
I20240501 20:01:16.918942 29819 tablet_copy_service.cc:140] P ee0d97247d394a8facfa774f15f3a67d: Received BeginTabletCopySession request for tablet 9046e38fc57f4364aa3d539e1fc01928 from peer 888d99522e4844118f8f1cf26f268c8c ({username='slave'} at 127.0.0.1:49474)
I20240501 20:01:16.919510 29819 tablet_copy_service.cc:161] P ee0d97247d394a8facfa774f15f3a67d: Beginning new tablet copy session on tablet 9046e38fc57f4364aa3d539e1fc01928 from peer 888d99522e4844118f8f1cf26f268c8c at {username='slave'} at 127.0.0.1:49474: session id = 888d99522e4844118f8f1cf26f268c8c-9046e38fc57f4364aa3d539e1fc01928
I20240501 20:01:16.938062 30479 ts_tablet_manager.cc:902] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c: Initiating tablet copy from peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959)
I20240501 20:01:16.943884 29819 tablet_copy_source_session.cc:205] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:01:16.963806 30490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9046e38fc57f4364aa3d539e1fc01928. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:16.968598 30479 tablet_copy_client.cc:308] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Beginning tablet copy session from remote peer at address 127.27.217.193:33959
I20240501 20:01:16.988430 30490 tablet_copy_client.cc:783] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Starting download of 0 data blocks...
I20240501 20:01:16.989122 30490 tablet_copy_client.cc:647] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Starting download of 1 WAL segments...
I20240501 20:01:16.995365 30490 tablet_copy_client.cc:523] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:01:17.004457 30494 tablet_copy_client.cc:308] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Beginning tablet copy session from remote peer at address 127.27.217.194:32907
I20240501 20:01:17.015286 29670 tablet_copy_service.cc:140] P 4c8da1d580614449a636b855bf11b95f: Received BeginTabletCopySession request for tablet 8bd5e83faaff4e2da4e4cc22ba410d33 from peer 888d99522e4844118f8f1cf26f268c8c ({username='slave'} at 127.0.0.1:37936)
I20240501 20:01:17.015817 29670 tablet_copy_service.cc:161] P 4c8da1d580614449a636b855bf11b95f: Beginning new tablet copy session on tablet 8bd5e83faaff4e2da4e4cc22ba410d33 from peer 888d99522e4844118f8f1cf26f268c8c at {username='slave'} at 127.0.0.1:37936: session id = 888d99522e4844118f8f1cf26f268c8c-8bd5e83faaff4e2da4e4cc22ba410d33
I20240501 20:01:17.020684 30490 tablet_bootstrap.cc:492] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c: Bootstrap starting.
I20240501 20:01:17.027220 29670 tablet_copy_source_session.cc:205] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:01:17.030658 30479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bd5e83faaff4e2da4e4cc22ba410d33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:17.048306 30479 tablet_copy_client.cc:783] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Starting download of 0 data blocks...
I20240501 20:01:17.049005 30479 tablet_copy_client.cc:647] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Starting download of 1 WAL segments...
I20240501 20:01:17.053563 30479 tablet_copy_client.cc:523] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:01:17.072801 30479 tablet_bootstrap.cc:492] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c: Bootstrap starting.
I20240501 20:01:17.076632 30420 raft_consensus.cc:1203] T add9d7579351483fb268f89a44e95d8a P 888d99522e4844118f8f1cf26f268c8c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:01:17.079828 29745 tablet_copy_service.cc:140] P fdfeac4190e045f290e003b2298d988d: Received BeginTabletCopySession request for tablet cb05df43bdfa479b902b1da297fb5974 from peer 888d99522e4844118f8f1cf26f268c8c ({username='slave'} at 127.0.0.1:47414)
I20240501 20:01:17.080538 29745 tablet_copy_service.cc:161] P fdfeac4190e045f290e003b2298d988d: Beginning new tablet copy session on tablet cb05df43bdfa479b902b1da297fb5974 from peer 888d99522e4844118f8f1cf26f268c8c at {username='slave'} at 127.0.0.1:47414: session id = 888d99522e4844118f8f1cf26f268c8c-cb05df43bdfa479b902b1da297fb5974
I20240501 20:01:17.103931 29745 tablet_copy_source_session.cc:205] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:01:17.107460 30494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb05df43bdfa479b902b1da297fb5974. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:17.138473 30490 tablet_bootstrap.cc:492] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:01:17.144830 30490 tablet_bootstrap.cc:492] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c: Bootstrap complete.
I20240501 20:01:17.145637 30490 ts_tablet_manager.cc:1367] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c: Time spent bootstrapping tablet: real 0.125s user 0.030s sys 0.009s
I20240501 20:01:17.148676 30490 raft_consensus.cc:348] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:17.149936 30490 raft_consensus.cc:729] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 888d99522e4844118f8f1cf26f268c8c, State: Initialized, Role: LEARNER
I20240501 20:01:17.150928 30490 consensus_queue.cc:260] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:17.153998 30490 ts_tablet_manager.cc:1398] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240501 20:01:17.154282 30494 tablet_copy_client.cc:783] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Starting download of 0 data blocks...
I20240501 20:01:17.154860 30494 tablet_copy_client.cc:647] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Starting download of 1 WAL segments...
I20240501 20:01:17.161772 29819 tablet_copy_service.cc:342] P ee0d97247d394a8facfa774f15f3a67d: Request end of tablet copy session 888d99522e4844118f8f1cf26f268c8c-9046e38fc57f4364aa3d539e1fc01928 received from {username='slave'} at 127.0.0.1:49474
I20240501 20:01:17.162240 29819 tablet_copy_service.cc:433] P ee0d97247d394a8facfa774f15f3a67d: ending tablet copy session 888d99522e4844118f8f1cf26f268c8c-9046e38fc57f4364aa3d539e1fc01928 on tablet 9046e38fc57f4364aa3d539e1fc01928 with peer 888d99522e4844118f8f1cf26f268c8c
I20240501 20:01:17.167114 30494 tablet_copy_client.cc:523] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:01:17.187376 30494 tablet_bootstrap.cc:492] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c: Bootstrap starting.
I20240501 20:01:17.198648 30479 tablet_bootstrap.cc:492] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:01:17.199611 30479 tablet_bootstrap.cc:492] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c: Bootstrap complete.
I20240501 20:01:17.200340 30479 ts_tablet_manager.cc:1367] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c: Time spent bootstrapping tablet: real 0.128s user 0.028s sys 0.002s
I20240501 20:01:17.203006 30479 raft_consensus.cc:348] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:17.218575 30479 raft_consensus.cc:729] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 888d99522e4844118f8f1cf26f268c8c, State: Initialized, Role: LEARNER
I20240501 20:01:17.219221 30479 consensus_queue.cc:260] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:17.236588 30479 ts_tablet_manager.cc:1398] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c: Time spent starting tablet: real 0.036s user 0.005s sys 0.001s
I20240501 20:01:17.237030 30494 tablet_bootstrap.cc:492] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:01:17.238011 30494 tablet_bootstrap.cc:492] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c: Bootstrap complete.
I20240501 20:01:17.238883 30494 ts_tablet_manager.cc:1367] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c: Time spent bootstrapping tablet: real 0.052s user 0.012s sys 0.008s
I20240501 20:01:17.242298 30494 raft_consensus.cc:348] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:17.243288 30494 raft_consensus.cc:729] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 888d99522e4844118f8f1cf26f268c8c, State: Initialized, Role: LEARNER
I20240501 20:01:17.244012 30494 consensus_queue.cc:260] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:17.248193 30494 ts_tablet_manager.cc:1398] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c: Time spent starting tablet: real 0.008s user 0.001s sys 0.005s
I20240501 20:01:17.249804 29745 tablet_copy_service.cc:342] P fdfeac4190e045f290e003b2298d988d: Request end of tablet copy session 888d99522e4844118f8f1cf26f268c8c-cb05df43bdfa479b902b1da297fb5974 received from {username='slave'} at 127.0.0.1:47414
I20240501 20:01:17.250200 29745 tablet_copy_service.cc:433] P fdfeac4190e045f290e003b2298d988d: ending tablet copy session 888d99522e4844118f8f1cf26f268c8c-cb05df43bdfa479b902b1da297fb5974 on tablet cb05df43bdfa479b902b1da297fb5974 with peer 888d99522e4844118f8f1cf26f268c8c
I20240501 20:01:17.262244 29670 tablet_copy_service.cc:342] P 4c8da1d580614449a636b855bf11b95f: Request end of tablet copy session 888d99522e4844118f8f1cf26f268c8c-8bd5e83faaff4e2da4e4cc22ba410d33 received from {username='slave'} at 127.0.0.1:37936
I20240501 20:01:17.262679 29670 tablet_copy_service.cc:433] P 4c8da1d580614449a636b855bf11b95f: ending tablet copy session 888d99522e4844118f8f1cf26f268c8c-8bd5e83faaff4e2da4e4cc22ba410d33 on tablet 8bd5e83faaff4e2da4e4cc22ba410d33 with peer 888d99522e4844118f8f1cf26f268c8c
I20240501 20:01:17.268515 30420 raft_consensus.cc:1203] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:01:17.481500 30420 raft_consensus.cc:1203] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:01:17.526947 30420 raft_consensus.cc:1203] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [term 3 LEARNER]: Deduplicated request from leader. Original: 3.2->[3.3-3.3] Dedup: 3.3->[]
I20240501 20:01:17.582798 30497 raft_consensus.cc:1050] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d: attempting to promote NON_VOTER 888d99522e4844118f8f1cf26f268c8c to VOTER
I20240501 20:01:17.585702 30497 consensus_queue.cc:237] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:17.596737 30420 raft_consensus.cc:1261] T add9d7579351483fb268f89a44e95d8a P 888d99522e4844118f8f1cf26f268c8c [term 1 LEARNER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:01:17.598315 30370 consensus_queue.cc:1035] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:17.600741 29652 raft_consensus.cc:1261] T add9d7579351483fb268f89a44e95d8a P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:01:17.608785 30370 consensus_queue.cc:1035] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20240501 20:01:17.598575 29732 raft_consensus.cc:1261] T add9d7579351483fb268f89a44e95d8a P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:01:17.615581 30370 consensus_queue.cc:1035] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:17.638741 30370 raft_consensus.cc:2932] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:17.653811 30420 raft_consensus.cc:2932] T add9d7579351483fb268f89a44e95d8a P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:17.657177 29652 raft_consensus.cc:2932] T add9d7579351483fb268f89a44e95d8a P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:17.657299 29733 raft_consensus.cc:2932] T add9d7579351483fb268f89a44e95d8a P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:17.673179 29554 catalog_manager.cc:5484] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d reported cstate change: config changed from index 2 to 3, 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:01:17.717417 29809 consensus_queue.cc:237] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:17.726740 30420 raft_consensus.cc:1261] T add9d7579351483fb268f89a44e95d8a P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:01:17.728157 30511 consensus_queue.cc:1035] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:01:17.729075 29653 raft_consensus.cc:1261] T add9d7579351483fb268f89a44e95d8a P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:01:17.735405 30497 consensus_queue.cc:1035] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:01:17.737447 30497 raft_consensus.cc:2932] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fdfeac4190e045f290e003b2298d988d (127.27.217.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:17.744177 30420 raft_consensus.cc:2932] T add9d7579351483fb268f89a44e95d8a P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fdfeac4190e045f290e003b2298d988d (127.27.217.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:17.748160 29544 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet add9d7579351483fb268f89a44e95d8a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:01:17.754307 29652 raft_consensus.cc:2932] T add9d7579351483fb268f89a44e95d8a P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fdfeac4190e045f290e003b2298d988d (127.27.217.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:17.765664 29557 catalog_manager.cc:5484] T add9d7579351483fb268f89a44e95d8a P 888d99522e4844118f8f1cf26f268c8c reported cstate change: config changed from index 3 to 4, VOTER fdfeac4190e045f290e003b2298d988d (127.27.217.194) evicted. New cstate: current_term: 1 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:17.789086 30512 raft_consensus.cc:1050] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d: attempting to promote NON_VOTER 888d99522e4844118f8f1cf26f268c8c to VOTER
I20240501 20:01:17.792213 29725 tablet_service.cc:1514] Processing DeleteTablet for tablet add9d7579351483fb268f89a44e95d8a with delete_type TABLET_DATA_TOMBSTONED (TS fdfeac4190e045f290e003b2298d988d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36258
I20240501 20:01:17.794518 30521 tablet_replica.cc:330] T add9d7579351483fb268f89a44e95d8a P fdfeac4190e045f290e003b2298d988d: stopping tablet replica
I20240501 20:01:17.791450 30512 consensus_queue.cc:237] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:17.795557 30521 raft_consensus.cc:2229] T add9d7579351483fb268f89a44e95d8a P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:17.796298 30521 raft_consensus.cc:2258] T add9d7579351483fb268f89a44e95d8a P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:17.799669 30521 ts_tablet_manager.cc:1875] T add9d7579351483fb268f89a44e95d8a P fdfeac4190e045f290e003b2298d988d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:01:17.822515 29726 raft_consensus.cc:1261] T 9046e38fc57f4364aa3d539e1fc01928 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:01:17.824084 30420 raft_consensus.cc:1261] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [term 1 LEARNER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:01:17.825644 29653 raft_consensus.cc:1261] T 9046e38fc57f4364aa3d539e1fc01928 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:01:17.827150 30511 consensus_queue.cc:1035] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:17.828316 30370 consensus_queue.cc:1035] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:17.829434 30512 consensus_queue.cc:1035] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:17.834893 30521 ts_tablet_manager.cc:1888] T add9d7579351483fb268f89a44e95d8a P fdfeac4190e045f290e003b2298d988d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:01:17.835440 30521 log.cc:1196] T add9d7579351483fb268f89a44e95d8a P fdfeac4190e045f290e003b2298d988d: Deleting WAL directory at /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714593602638719-28519-0/minicluster-data/ts-1-root/wals/add9d7579351483fb268f89a44e95d8a
I20240501 20:01:17.838191 29543 catalog_manager.cc:4830] TS fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): tablet add9d7579351483fb268f89a44e95d8a (table test-workload [id=c4cd6efa2a67402dac287d449c66135b]) successfully deleted
I20240501 20:01:17.864310 30526 raft_consensus.cc:1050] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d: attempting to promote NON_VOTER 888d99522e4844118f8f1cf26f268c8c to VOTER
I20240501 20:01:17.874533 30526 consensus_queue.cc:237] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:17.877689 30512 raft_consensus.cc:2932] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:17.880698 30420 raft_consensus.cc:2932] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:17.898505 29558 catalog_manager.cc:5484] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c reported cstate change: config changed from index 2 to 3, 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:17.899365 29730 raft_consensus.cc:2932] T 9046e38fc57f4364aa3d539e1fc01928 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:17.916597 29652 raft_consensus.cc:2932] T 9046e38fc57f4364aa3d539e1fc01928 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:17.934774 29809 consensus_queue.cc:237] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:17.947158 30420 raft_consensus.cc:1261] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:01:17.948655 30458 consensus_queue.cc:1035] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240501 20:01:17.952260 29809 raft_consensus.cc:1261] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:01:17.954627 30279 consensus_queue.cc:1035] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 20:01:17.965623 30419 raft_consensus.cc:1261] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [term 1 LEARNER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:01:17.970134 29656 raft_consensus.cc:1261] T cb05df43bdfa479b902b1da297fb5974 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:01:17.974382 30370 raft_consensus.cc:2932] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4c8da1d580614449a636b855bf11b95f (127.27.217.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:17.977855 30419 raft_consensus.cc:2932] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4c8da1d580614449a636b855bf11b95f (127.27.217.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:17.988741 29730 raft_consensus.cc:1261] T 9046e38fc57f4364aa3d539e1fc01928 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:01:18.003923 30497 consensus_queue.cc:1035] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240501 20:01:18.014330 29544 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9046e38fc57f4364aa3d539e1fc01928 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:01:18.015934 29730 raft_consensus.cc:2932] T 9046e38fc57f4364aa3d539e1fc01928 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4c8da1d580614449a636b855bf11b95f (127.27.217.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:18.020345 30481 consensus_queue.cc:1035] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 20:01:18.024184 30528 consensus_queue.cc:1035] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 20:01:18.039935 29557 catalog_manager.cc:5484] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c reported cstate change: config changed from index 3 to 4, VOTER 4c8da1d580614449a636b855bf11b95f (127.27.217.193) evicted. New cstate: current_term: 1 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:18.054908 29650 tablet_service.cc:1514] Processing DeleteTablet for tablet 9046e38fc57f4364aa3d539e1fc01928 with delete_type TABLET_DATA_TOMBSTONED (TS 4c8da1d580614449a636b855bf11b95f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59124
I20240501 20:01:18.078608 30532 tablet_replica.cc:330] T 9046e38fc57f4364aa3d539e1fc01928 P 4c8da1d580614449a636b855bf11b95f: stopping tablet replica
I20240501 20:01:18.085443 30528 raft_consensus.cc:2932] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:18.095000 30302 raft_consensus.cc:1050] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f: attempting to promote NON_VOTER 888d99522e4844118f8f1cf26f268c8c to VOTER
I20240501 20:01:18.097339 30302 consensus_queue.cc:237] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 1, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:18.091081 30532 raft_consensus.cc:2229] T 9046e38fc57f4364aa3d539e1fc01928 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:18.101367 29656 raft_consensus.cc:2932] T cb05df43bdfa479b902b1da297fb5974 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:18.102972 30532 raft_consensus.cc:2258] T 9046e38fc57f4364aa3d539e1fc01928 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:18.106477 29800 raft_consensus.cc:2932] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:18.118052 29554 catalog_manager.cc:5484] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d reported cstate change: config changed from index 2 to 3, 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:18.123682 30532 ts_tablet_manager.cc:1875] T 9046e38fc57f4364aa3d539e1fc01928 P 4c8da1d580614449a636b855bf11b95f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:01:18.132788 30420 raft_consensus.cc:1261] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [term 3 LEARNER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240501 20:01:18.138360 29800 raft_consensus.cc:1261] T 8bd5e83faaff4e2da4e4cc22ba410d33 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240501 20:01:18.136298 30488 consensus_queue.cc:1035] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20240501 20:01:18.141299 30488 consensus_queue.cc:1035] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240501 20:01:18.151952 29730 raft_consensus.cc:1261] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240501 20:01:18.163376 30488 consensus_queue.cc:1035] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20240501 20:01:18.191051 30302 raft_consensus.cc:2932] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:18.202198 29732 consensus_queue.cc:237] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:18.204017 29730 raft_consensus.cc:2932] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:18.210356 30418 raft_consensus.cc:2932] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:18.220342 29803 raft_consensus.cc:1261] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:01:18.217875 30420 raft_consensus.cc:2932] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:18.254763 29800 raft_consensus.cc:2932] T 8bd5e83faaff4e2da4e4cc22ba410d33 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:18.258591 30526 consensus_queue.cc:1035] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:01:18.273739 29556 catalog_manager.cc:5484] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f reported cstate change: config changed from index 3 to 4, 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:01:18.291378 30419 raft_consensus.cc:1261] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:01:18.294108 30481 raft_consensus.cc:2932] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4c8da1d580614449a636b855bf11b95f (127.27.217.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:18.300066 30528 consensus_queue.cc:1035] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:01:18.304958 30532 ts_tablet_manager.cc:1888] T 9046e38fc57f4364aa3d539e1fc01928 P 4c8da1d580614449a636b855bf11b95f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:01:18.305471 30532 log.cc:1196] T 9046e38fc57f4364aa3d539e1fc01928 P 4c8da1d580614449a636b855bf11b95f: Deleting WAL directory at /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714593602638719-28519-0/minicluster-data/ts-0-root/wals/9046e38fc57f4364aa3d539e1fc01928
I20240501 20:01:18.305691 30420 raft_consensus.cc:2932] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4c8da1d580614449a636b855bf11b95f (127.27.217.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:18.307379 29803 raft_consensus.cc:2932] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4c8da1d580614449a636b855bf11b95f (127.27.217.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:18.308501 29543 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cb05df43bdfa479b902b1da297fb5974 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:01:18.319928 29542 catalog_manager.cc:4830] TS 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): tablet 9046e38fc57f4364aa3d539e1fc01928 (table test-workload [id=c4cd6efa2a67402dac287d449c66135b]) successfully deleted
I20240501 20:01:18.330400 29655 consensus_queue.cc:237] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:18.333349 29557 catalog_manager.cc:5484] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c reported cstate change: config changed from index 3 to 4, VOTER 4c8da1d580614449a636b855bf11b95f (127.27.217.193) evicted. New cstate: current_term: 1 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:18.348740 30420 raft_consensus.cc:1261] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240501 20:01:18.351425 30488 consensus_queue.cc:1035] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:01:18.342931 29732 raft_consensus.cc:1261] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240501 20:01:18.358332 30538 consensus_queue.cc:1035] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:01:18.362370 29650 tablet_service.cc:1514] Processing DeleteTablet for tablet cb05df43bdfa479b902b1da297fb5974 with delete_type TABLET_DATA_TOMBSTONED (TS 4c8da1d580614449a636b855bf11b95f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59124
I20240501 20:01:18.363348 30532 tablet_replica.cc:330] T cb05df43bdfa479b902b1da297fb5974 P 4c8da1d580614449a636b855bf11b95f: stopping tablet replica
I20240501 20:01:18.364786 30532 raft_consensus.cc:2229] T cb05df43bdfa479b902b1da297fb5974 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:18.365476 30532 raft_consensus.cc:2258] T cb05df43bdfa479b902b1da297fb5974 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:18.369251 30532 ts_tablet_manager.cc:1875] T cb05df43bdfa479b902b1da297fb5974 P 4c8da1d580614449a636b855bf11b95f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:01:18.369684 30534 raft_consensus.cc:2932] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:18.376965 30420 raft_consensus.cc:2932] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:18.393139 29542 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8bd5e83faaff4e2da4e4cc22ba410d33 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:01:18.398797 30532 ts_tablet_manager.cc:1888] T cb05df43bdfa479b902b1da297fb5974 P 4c8da1d580614449a636b855bf11b95f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:01:18.401201 30532 log.cc:1196] T cb05df43bdfa479b902b1da297fb5974 P 4c8da1d580614449a636b855bf11b95f: Deleting WAL directory at /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714593602638719-28519-0/minicluster-data/ts-0-root/wals/cb05df43bdfa479b902b1da297fb5974
I20240501 20:01:18.403095 29542 catalog_manager.cc:4830] TS 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): tablet cb05df43bdfa479b902b1da297fb5974 (table test-workload [id=c4cd6efa2a67402dac287d449c66135b]) successfully deleted
I20240501 20:01:18.397981 29731 raft_consensus.cc:2932] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:18.405552 29558 catalog_manager.cc:5484] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f reported cstate change: config changed from index 4 to 5, VOTER ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) evicted. New cstate: current_term: 3 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:01:18.417363 29793 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bd5e83faaff4e2da4e4cc22ba410d33 with delete_type TABLET_DATA_TOMBSTONED (TS ee0d97247d394a8facfa774f15f3a67d not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41856
I20240501 20:01:18.429081 30539 tablet_replica.cc:330] T 8bd5e83faaff4e2da4e4cc22ba410d33 P ee0d97247d394a8facfa774f15f3a67d: stopping tablet replica
I20240501 20:01:18.430155 30539 raft_consensus.cc:2229] T 8bd5e83faaff4e2da4e4cc22ba410d33 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:18.438647 30539 raft_consensus.cc:2258] T 8bd5e83faaff4e2da4e4cc22ba410d33 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:18.446588 30539 ts_tablet_manager.cc:1875] T 8bd5e83faaff4e2da4e4cc22ba410d33 P ee0d97247d394a8facfa774f15f3a67d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:01:18.469594 30539 ts_tablet_manager.cc:1888] T 8bd5e83faaff4e2da4e4cc22ba410d33 P ee0d97247d394a8facfa774f15f3a67d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.4
I20240501 20:01:18.469892 30539 log.cc:1196] T 8bd5e83faaff4e2da4e4cc22ba410d33 P ee0d97247d394a8facfa774f15f3a67d: Deleting WAL directory at /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714593602638719-28519-0/minicluster-data/ts-2-root/wals/8bd5e83faaff4e2da4e4cc22ba410d33
I20240501 20:01:18.471233 29544 catalog_manager.cc:4830] TS ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): tablet 8bd5e83faaff4e2da4e4cc22ba410d33 (table test-workload [id=c4cd6efa2a67402dac287d449c66135b]) successfully deleted
I20240501 20:01:19.471450 29655 consensus_queue.cc:237] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:19.479182 29803 raft_consensus.cc:1261] T 25e07220db7444ae97f294d70eda6a93 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:01:19.479323 29623 consensus_peers.cc:485] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f -> Peer 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227): Couldn't send request to peer 888d99522e4844118f8f1cf26f268c8c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 25e07220db7444ae97f294d70eda6a93. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:19.480692 29732 raft_consensus.cc:1261] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:01:19.480473 30538 consensus_queue.cc:1035] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:19.482147 30538 consensus_queue.cc:1035] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:19.493392 30534 raft_consensus.cc:2932] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
I20240501 20:01:19.496809 29803 raft_consensus.cc:2932] T 25e07220db7444ae97f294d70eda6a93 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
I20240501 20:01:19.504516 29731 raft_consensus.cc:2932] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
I20240501 20:01:19.508134 29655 consensus_queue.cc:237] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:19.515496 29557 catalog_manager.cc:5484] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f reported cstate change: config changed from index -1 to 2, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New cstate: current_term: 1 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 20:01:19.526116 29623 consensus_peers.cc:485] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f -> Peer 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227): Couldn't send request to peer 888d99522e4844118f8f1cf26f268c8c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fa7e89c0572e4ee7be8e5315b459bdb3. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:19.528712 29803 raft_consensus.cc:1261] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:01:19.529310 29732 raft_consensus.cc:1261] T fa7e89c0572e4ee7be8e5315b459bdb3 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:01:19.530074 30534 consensus_queue.cc:1035] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:19.535893 30538 consensus_queue.cc:1035] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 20:01:19.579435 30488 raft_consensus.cc:2932] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
I20240501 20:01:19.581521 29809 raft_consensus.cc:2932] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
I20240501 20:01:19.625049 29731 raft_consensus.cc:2932] T fa7e89c0572e4ee7be8e5315b459bdb3 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
I20240501 20:01:19.632179 29655 consensus_queue.cc:237] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:19.632490 29554 catalog_manager.cc:5484] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d reported cstate change: config changed from index -1 to 2, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New cstate: current_term: 1 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
I20240501 20:01:19.650964 29809 raft_consensus.cc:1261] T 7ed27033ab1645e9ab0600ab635d9589 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:01:19.651170 29623 consensus_peers.cc:485] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f -> Peer 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227): Couldn't send request to peer 888d99522e4844118f8f1cf26f268c8c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ed27033ab1645e9ab0600ab635d9589. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:19.651690 29732 raft_consensus.cc:1261] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:01:19.652411 30553 consensus_queue.cc:1035] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:19.653604 30551 consensus_queue.cc:1035] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:19.671959 30553 raft_consensus.cc:2932] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
I20240501 20:01:19.674881 29800 raft_consensus.cc:2932] T 7ed27033ab1645e9ab0600ab635d9589 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
I20240501 20:01:19.702036 29803 consensus_queue.cc:237] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:19.707309 29554 catalog_manager.cc:5484] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f reported cstate change: config changed from index -1 to 2, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New cstate: current_term: 1 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:19.707454 29726 raft_consensus.cc:2932] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
W20240501 20:01:19.738494 29772 consensus_peers.cc:485] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d -> Peer 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227): Couldn't send request to peer 888d99522e4844118f8f1cf26f268c8c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 696121417ea9403990456cfc6fc4f416. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:19.742411 29726 raft_consensus.cc:1261] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:01:19.744597 29655 raft_consensus.cc:1261] T 696121417ea9403990456cfc6fc4f416 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:01:19.746425 30497 consensus_queue.cc:1035] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:19.757521 30458 consensus_queue.cc:1035] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:19.769838 29653 raft_consensus.cc:2932] T 696121417ea9403990456cfc6fc4f416 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
I20240501 20:01:19.769258 30497 raft_consensus.cc:2932] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
I20240501 20:01:19.775025 29732 raft_consensus.cc:2932] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
I20240501 20:01:19.803047 29554 catalog_manager.cc:5484] T 696121417ea9403990456cfc6fc4f416 P 4c8da1d580614449a636b855bf11b95f reported cstate change: config changed from index -1 to 2, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New cstate: current_term: 1 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
I20240501 20:01:20.019292 30562 ts_tablet_manager.cc:902] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c: Initiating tablet copy from peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959)
I20240501 20:01:20.033365 30562 tablet_copy_client.cc:308] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Beginning tablet copy session from remote peer at address 127.27.217.193:33959
I20240501 20:01:20.035161 29670 tablet_copy_service.cc:140] P 4c8da1d580614449a636b855bf11b95f: Received BeginTabletCopySession request for tablet 25e07220db7444ae97f294d70eda6a93 from peer 888d99522e4844118f8f1cf26f268c8c ({username='slave'} at 127.0.0.1:37936)
I20240501 20:01:20.035720 29670 tablet_copy_service.cc:161] P 4c8da1d580614449a636b855bf11b95f: Beginning new tablet copy session on tablet 25e07220db7444ae97f294d70eda6a93 from peer 888d99522e4844118f8f1cf26f268c8c at {username='slave'} at 127.0.0.1:37936: session id = 888d99522e4844118f8f1cf26f268c8c-25e07220db7444ae97f294d70eda6a93
I20240501 20:01:20.044428 29670 tablet_copy_source_session.cc:205] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:01:20.049443 30562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25e07220db7444ae97f294d70eda6a93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:20.069550 30562 tablet_copy_client.cc:783] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Starting download of 0 data blocks...
I20240501 20:01:20.070148 30562 tablet_copy_client.cc:647] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Starting download of 1 WAL segments...
I20240501 20:01:20.074704 30562 tablet_copy_client.cc:523] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:01:20.083848 30562 tablet_bootstrap.cc:492] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c: Bootstrap starting.
I20240501 20:01:20.102597 30562 tablet_bootstrap.cc:492] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:01:20.103438 30562 tablet_bootstrap.cc:492] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c: Bootstrap complete.
I20240501 20:01:20.104107 30562 ts_tablet_manager.cc:1367] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c: Time spent bootstrapping tablet: real 0.020s user 0.023s sys 0.000s
I20240501 20:01:20.108177 30562 raft_consensus.cc:348] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:20.110062 30562 raft_consensus.cc:729] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 888d99522e4844118f8f1cf26f268c8c, State: Initialized, Role: LEARNER
I20240501 20:01:20.110738 30562 consensus_queue.cc:260] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:20.116899 30562 ts_tablet_manager.cc:1398] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c: Time spent starting tablet: real 0.013s user 0.005s sys 0.004s
I20240501 20:01:20.119272 29670 tablet_copy_service.cc:342] P 4c8da1d580614449a636b855bf11b95f: Request end of tablet copy session 888d99522e4844118f8f1cf26f268c8c-25e07220db7444ae97f294d70eda6a93 received from {username='slave'} at 127.0.0.1:37936
I20240501 20:01:20.119704 29670 tablet_copy_service.cc:433] P 4c8da1d580614449a636b855bf11b95f: ending tablet copy session 888d99522e4844118f8f1cf26f268c8c-25e07220db7444ae97f294d70eda6a93 on tablet 25e07220db7444ae97f294d70eda6a93 with peer 888d99522e4844118f8f1cf26f268c8c
I20240501 20:01:20.142788 30562 ts_tablet_manager.cc:902] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c: Initiating tablet copy from peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959)
I20240501 20:01:20.185521 30570 ts_tablet_manager.cc:902] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c: Initiating tablet copy from peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:20.209408 30562 tablet_copy_client.cc:308] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Beginning tablet copy session from remote peer at address 127.27.217.193:33959
I20240501 20:01:20.213639 29670 tablet_copy_service.cc:140] P 4c8da1d580614449a636b855bf11b95f: Received BeginTabletCopySession request for tablet fa7e89c0572e4ee7be8e5315b459bdb3 from peer 888d99522e4844118f8f1cf26f268c8c ({username='slave'} at 127.0.0.1:37936)
I20240501 20:01:20.214517 29670 tablet_copy_service.cc:161] P 4c8da1d580614449a636b855bf11b95f: Beginning new tablet copy session on tablet fa7e89c0572e4ee7be8e5315b459bdb3 from peer 888d99522e4844118f8f1cf26f268c8c at {username='slave'} at 127.0.0.1:37936: session id = 888d99522e4844118f8f1cf26f268c8c-fa7e89c0572e4ee7be8e5315b459bdb3
I20240501 20:01:20.220311 29670 tablet_copy_source_session.cc:205] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:01:20.220865 30570 tablet_copy_client.cc:308] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Beginning tablet copy session from remote peer at address 127.27.217.195:35861
I20240501 20:01:20.222265 29819 tablet_copy_service.cc:140] P ee0d97247d394a8facfa774f15f3a67d: Received BeginTabletCopySession request for tablet 696121417ea9403990456cfc6fc4f416 from peer 888d99522e4844118f8f1cf26f268c8c ({username='slave'} at 127.0.0.1:49474)
I20240501 20:01:20.222641 29819 tablet_copy_service.cc:161] P ee0d97247d394a8facfa774f15f3a67d: Beginning new tablet copy session on tablet 696121417ea9403990456cfc6fc4f416 from peer 888d99522e4844118f8f1cf26f268c8c at {username='slave'} at 127.0.0.1:49474: session id = 888d99522e4844118f8f1cf26f268c8c-696121417ea9403990456cfc6fc4f416
I20240501 20:01:20.229524 30562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa7e89c0572e4ee7be8e5315b459bdb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:20.235451 29819 tablet_copy_source_session.cc:205] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:01:20.238662 30570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 696121417ea9403990456cfc6fc4f416. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:20.254599 30562 tablet_copy_client.cc:783] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Starting download of 0 data blocks...
I20240501 20:01:20.254597 30570 tablet_copy_client.cc:783] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Starting download of 0 data blocks...
I20240501 20:01:20.255367 30562 tablet_copy_client.cc:647] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Starting download of 1 WAL segments...
I20240501 20:01:20.255434 30570 tablet_copy_client.cc:647] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Starting download of 1 WAL segments...
I20240501 20:01:20.260951 30570 tablet_copy_client.cc:523] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:01:20.262689 30562 tablet_copy_client.cc:523] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:01:20.278587 30562 tablet_bootstrap.cc:492] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c: Bootstrap starting.
I20240501 20:01:20.297820 30570 tablet_bootstrap.cc:492] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c: Bootstrap starting.
I20240501 20:01:20.301393 30578 ts_tablet_manager.cc:902] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c: Initiating tablet copy from peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959)
W20240501 20:01:20.361300 30562 log.cc:924] Time spent T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c: Append to log took a long time: real 0.055s user 0.001s sys 0.000s
I20240501 20:01:20.370273 30562 tablet_bootstrap.cc:492] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:01:20.371196 30562 tablet_bootstrap.cc:492] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c: Bootstrap complete.
I20240501 20:01:20.397188 30578 tablet_copy_client.cc:308] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Beginning tablet copy session from remote peer at address 127.27.217.193:33959
I20240501 20:01:20.371922 30562 ts_tablet_manager.cc:1367] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c: Time spent bootstrapping tablet: real 0.094s user 0.022s sys 0.007s
I20240501 20:01:20.398888 29670 tablet_copy_service.cc:140] P 4c8da1d580614449a636b855bf11b95f: Received BeginTabletCopySession request for tablet 7ed27033ab1645e9ab0600ab635d9589 from peer 888d99522e4844118f8f1cf26f268c8c ({username='slave'} at 127.0.0.1:37936)
I20240501 20:01:20.399422 29670 tablet_copy_service.cc:161] P 4c8da1d580614449a636b855bf11b95f: Beginning new tablet copy session on tablet 7ed27033ab1645e9ab0600ab635d9589 from peer 888d99522e4844118f8f1cf26f268c8c at {username='slave'} at 127.0.0.1:37936: session id = 888d99522e4844118f8f1cf26f268c8c-7ed27033ab1645e9ab0600ab635d9589
I20240501 20:01:20.402194 30562 raft_consensus.cc:348] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:20.405088 30562 raft_consensus.cc:729] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 888d99522e4844118f8f1cf26f268c8c, State: Initialized, Role: LEARNER
I20240501 20:01:20.405892 30562 consensus_queue.cc:260] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:20.407044 29670 tablet_copy_source_session.cc:205] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:01:20.413915 30420 raft_consensus.cc:1203] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:01:20.419883 30578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ed27033ab1645e9ab0600ab635d9589. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:20.407740 30570 tablet_bootstrap.cc:492] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:01:20.422623 30570 tablet_bootstrap.cc:492] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c: Bootstrap complete.
I20240501 20:01:20.423337 30570 ts_tablet_manager.cc:1367] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c: Time spent bootstrapping tablet: real 0.126s user 0.035s sys 0.025s
I20240501 20:01:20.428829 30570 raft_consensus.cc:348] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:20.429525 30570 raft_consensus.cc:729] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 888d99522e4844118f8f1cf26f268c8c, State: Initialized, Role: LEARNER
I20240501 20:01:20.430044 30570 consensus_queue.cc:260] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:20.432163 30570 ts_tablet_manager.cc:1398] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20240501 20:01:20.433969 29819 tablet_copy_service.cc:342] P ee0d97247d394a8facfa774f15f3a67d: Request end of tablet copy session 888d99522e4844118f8f1cf26f268c8c-696121417ea9403990456cfc6fc4f416 received from {username='slave'} at 127.0.0.1:49474
I20240501 20:01:20.434386 29819 tablet_copy_service.cc:433] P ee0d97247d394a8facfa774f15f3a67d: ending tablet copy session 888d99522e4844118f8f1cf26f268c8c-696121417ea9403990456cfc6fc4f416 on tablet 696121417ea9403990456cfc6fc4f416 with peer 888d99522e4844118f8f1cf26f268c8c
I20240501 20:01:20.408918 30562 ts_tablet_manager.cc:1398] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c: Time spent starting tablet: real 0.010s user 0.002s sys 0.004s
I20240501 20:01:20.459313 29670 tablet_copy_service.cc:342] P 4c8da1d580614449a636b855bf11b95f: Request end of tablet copy session 888d99522e4844118f8f1cf26f268c8c-fa7e89c0572e4ee7be8e5315b459bdb3 received from {username='slave'} at 127.0.0.1:37936
I20240501 20:01:20.459779 29670 tablet_copy_service.cc:433] P 4c8da1d580614449a636b855bf11b95f: ending tablet copy session 888d99522e4844118f8f1cf26f268c8c-fa7e89c0572e4ee7be8e5315b459bdb3 on tablet fa7e89c0572e4ee7be8e5315b459bdb3 with peer 888d99522e4844118f8f1cf26f268c8c
I20240501 20:01:20.485118 30578 tablet_copy_client.cc:783] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Starting download of 0 data blocks...
I20240501 20:01:20.485823 30578 tablet_copy_client.cc:647] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Starting download of 1 WAL segments...
I20240501 20:01:20.489317 30578 tablet_copy_client.cc:523] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:01:20.499366 30578 tablet_bootstrap.cc:492] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c: Bootstrap starting.
I20240501 20:01:20.528928 30578 tablet_bootstrap.cc:492] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:01:20.529851 30578 tablet_bootstrap.cc:492] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c: Bootstrap complete.
I20240501 20:01:20.530542 30578 ts_tablet_manager.cc:1367] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c: Time spent bootstrapping tablet: real 0.031s user 0.018s sys 0.011s
I20240501 20:01:20.533563 30578 raft_consensus.cc:348] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:20.534546 30578 raft_consensus.cc:729] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 888d99522e4844118f8f1cf26f268c8c, State: Initialized, Role: LEARNER
I20240501 20:01:20.535221 30578 consensus_queue.cc:260] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:20.538254 30578 ts_tablet_manager.cc:1398] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240501 20:01:20.543464 29670 tablet_copy_service.cc:342] P 4c8da1d580614449a636b855bf11b95f: Request end of tablet copy session 888d99522e4844118f8f1cf26f268c8c-7ed27033ab1645e9ab0600ab635d9589 received from {username='slave'} at 127.0.0.1:37936
I20240501 20:01:20.543921 29670 tablet_copy_service.cc:433] P 4c8da1d580614449a636b855bf11b95f: ending tablet copy session 888d99522e4844118f8f1cf26f268c8c-7ed27033ab1645e9ab0600ab635d9589 on tablet 7ed27033ab1645e9ab0600ab635d9589 with peer 888d99522e4844118f8f1cf26f268c8c
I20240501 20:01:20.636967 30420 raft_consensus.cc:1203] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:01:20.703569 30420 raft_consensus.cc:1203] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:01:20.861726 30551 raft_consensus.cc:1050] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f: attempting to promote NON_VOTER 888d99522e4844118f8f1cf26f268c8c to VOTER
I20240501 20:01:20.862586 30420 raft_consensus.cc:1203] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:01:20.863905 30551 consensus_queue.cc:237] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:20.871316 30420 raft_consensus.cc:1261] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [term 1 LEARNER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:01:20.871435 29809 raft_consensus.cc:1261] T 25e07220db7444ae97f294d70eda6a93 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:01:20.872018 29732 raft_consensus.cc:1261] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:01:20.873744 30551 consensus_queue.cc:1035] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.007s
I20240501 20:01:20.880987 30488 consensus_queue.cc:1035] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:20.881974 30553 consensus_queue.cc:1035] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:20.904423 30420 raft_consensus.cc:2932] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:20.906580 29809 raft_consensus.cc:2932] T 25e07220db7444ae97f294d70eda6a93 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:20.902922 30551 raft_consensus.cc:2932] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:20.926060 29558 catalog_manager.cc:5484] T 25e07220db7444ae97f294d70eda6a93 P ee0d97247d394a8facfa774f15f3a67d reported cstate change: config changed from index 2 to 3, 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:20.933744 29732 raft_consensus.cc:2932] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:20.978667 29657 consensus_queue.cc:237] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:20.984759 30420 raft_consensus.cc:1261] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:01:20.985210 29732 raft_consensus.cc:1261] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:01:20.986411 30551 consensus_queue.cc:1035] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:01:20.989357 30551 consensus_queue.cc:1035] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:01:20.996724 30589 raft_consensus.cc:2932] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:21.006197 29731 raft_consensus.cc:2932] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:21.014608 29542 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 25e07220db7444ae97f294d70eda6a93 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:01:21.015157 30420 raft_consensus.cc:2932] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:21.023974 29556 catalog_manager.cc:5484] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f reported cstate change: config changed from index 3 to 4, VOTER ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) evicted. New cstate: current_term: 1 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:01:21.042892 29793 tablet_service.cc:1514] Processing DeleteTablet for tablet 25e07220db7444ae97f294d70eda6a93 with delete_type TABLET_DATA_TOMBSTONED (TS ee0d97247d394a8facfa774f15f3a67d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41856
I20240501 20:01:21.045215 30597 tablet_replica.cc:330] T 25e07220db7444ae97f294d70eda6a93 P ee0d97247d394a8facfa774f15f3a67d: stopping tablet replica
I20240501 20:01:21.046087 30597 raft_consensus.cc:2229] T 25e07220db7444ae97f294d70eda6a93 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:21.046695 30597 raft_consensus.cc:2258] T 25e07220db7444ae97f294d70eda6a93 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:21.050069 30597 ts_tablet_manager.cc:1875] T 25e07220db7444ae97f294d70eda6a93 P ee0d97247d394a8facfa774f15f3a67d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:01:21.062655 30553 raft_consensus.cc:1050] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f: attempting to promote NON_VOTER 888d99522e4844118f8f1cf26f268c8c to VOTER
I20240501 20:01:21.064498 30597 ts_tablet_manager.cc:1888] T 25e07220db7444ae97f294d70eda6a93 P ee0d97247d394a8facfa774f15f3a67d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:01:21.064935 30597 log.cc:1196] T 25e07220db7444ae97f294d70eda6a93 P ee0d97247d394a8facfa774f15f3a67d: Deleting WAL directory at /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714593602638719-28519-0/minicluster-data/ts-2-root/wals/25e07220db7444ae97f294d70eda6a93
I20240501 20:01:21.065296 30553 consensus_queue.cc:237] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:21.068420 29544 catalog_manager.cc:4830] TS ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): tablet 25e07220db7444ae97f294d70eda6a93 (table test-workload [id=c4cd6efa2a67402dac287d449c66135b]) successfully deleted
I20240501 20:01:21.071346 30420 raft_consensus.cc:1261] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 1 LEARNER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:01:21.071756 29803 raft_consensus.cc:1261] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:01:21.072491 30551 consensus_queue.cc:1035] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:21.072985 29731 raft_consensus.cc:1261] T fa7e89c0572e4ee7be8e5315b459bdb3 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:01:21.073457 30488 consensus_queue.cc:1035] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:21.074476 30588 consensus_queue.cc:1035] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:21.097402 30551 raft_consensus.cc:2932] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:21.099272 29731 raft_consensus.cc:2932] T fa7e89c0572e4ee7be8e5315b459bdb3 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:21.106527 30420 raft_consensus.cc:2932] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:21.113646 29556 catalog_manager.cc:5484] T fa7e89c0572e4ee7be8e5315b459bdb3 P fdfeac4190e045f290e003b2298d988d reported cstate change: config changed from index 2 to 3, 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:21.117153 29803 raft_consensus.cc:2932] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:21.133896 29655 consensus_queue.cc:237] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:21.140936 29803 raft_consensus.cc:1261] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:01:21.142756 30553 consensus_queue.cc:1035] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:01:21.144016 30420 raft_consensus.cc:1261] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:01:21.148698 30488 raft_consensus.cc:2932] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fdfeac4190e045f290e003b2298d988d (127.27.217.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:21.150441 30551 consensus_queue.cc:1035] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:01:21.150328 29809 raft_consensus.cc:2932] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fdfeac4190e045f290e003b2298d988d (127.27.217.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:21.154549 30420 raft_consensus.cc:2932] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fdfeac4190e045f290e003b2298d988d (127.27.217.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:21.164407 29542 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fa7e89c0572e4ee7be8e5315b459bdb3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:01:21.170701 29556 catalog_manager.cc:5484] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d reported cstate change: config changed from index 3 to 4, VOTER fdfeac4190e045f290e003b2298d988d (127.27.217.194) evicted. New cstate: current_term: 1 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:21.182547 30586 raft_consensus.cc:1050] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d: attempting to promote NON_VOTER 888d99522e4844118f8f1cf26f268c8c to VOTER
I20240501 20:01:21.184568 30586 consensus_queue.cc:237] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:21.192190 30420 raft_consensus.cc:1261] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [term 1 LEARNER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:01:21.192904 29731 raft_consensus.cc:1261] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:01:21.194648 30458 consensus_queue.cc:1035] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:21.196262 30458 consensus_queue.cc:1035] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:21.204638 29655 raft_consensus.cc:1261] T 696121417ea9403990456cfc6fc4f416 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:01:21.205080 29725 tablet_service.cc:1514] Processing DeleteTablet for tablet fa7e89c0572e4ee7be8e5315b459bdb3 with delete_type TABLET_DATA_TOMBSTONED (TS fdfeac4190e045f290e003b2298d988d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36258
I20240501 20:01:21.207871 30497 consensus_queue.cc:1035] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:21.222420 30604 tablet_replica.cc:330] T fa7e89c0572e4ee7be8e5315b459bdb3 P fdfeac4190e045f290e003b2298d988d: stopping tablet replica
I20240501 20:01:21.223865 30604 raft_consensus.cc:2229] T fa7e89c0572e4ee7be8e5315b459bdb3 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:21.224737 30604 raft_consensus.cc:2258] T fa7e89c0572e4ee7be8e5315b459bdb3 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:21.229990 30604 ts_tablet_manager.cc:1875] T fa7e89c0572e4ee7be8e5315b459bdb3 P fdfeac4190e045f290e003b2298d988d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:01:21.238623 30603 raft_consensus.cc:2932] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:21.241302 29730 raft_consensus.cc:2932] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:21.252833 29655 raft_consensus.cc:2932] T 696121417ea9403990456cfc6fc4f416 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:21.260066 30604 ts_tablet_manager.cc:1888] T fa7e89c0572e4ee7be8e5315b459bdb3 P fdfeac4190e045f290e003b2298d988d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:01:21.260536 30604 log.cc:1196] T fa7e89c0572e4ee7be8e5315b459bdb3 P fdfeac4190e045f290e003b2298d988d: Deleting WAL directory at /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714593602638719-28519-0/minicluster-data/ts-1-root/wals/fa7e89c0572e4ee7be8e5315b459bdb3
I20240501 20:01:21.263813 29543 catalog_manager.cc:4830] TS fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): tablet fa7e89c0572e4ee7be8e5315b459bdb3 (table test-workload [id=c4cd6efa2a67402dac287d449c66135b]) successfully deleted
I20240501 20:01:21.269181 30420 raft_consensus.cc:2932] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:21.275099 29556 catalog_manager.cc:5484] T 696121417ea9403990456cfc6fc4f416 P 4c8da1d580614449a636b855bf11b95f reported cstate change: config changed from index 2 to 3, 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:21.314690 29803 consensus_queue.cc:237] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:21.318814 29730 raft_consensus.cc:1261] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:01:21.318853 30420 raft_consensus.cc:1261] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:01:21.320817 30603 consensus_queue.cc:1035] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:01:21.321616 30586 consensus_queue.cc:1035] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:01:21.330716 29730 raft_consensus.cc:2932] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4c8da1d580614449a636b855bf11b95f (127.27.217.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:21.331681 30603 raft_consensus.cc:2932] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4c8da1d580614449a636b855bf11b95f (127.27.217.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:21.332341 30420 raft_consensus.cc:2932] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4c8da1d580614449a636b855bf11b95f (127.27.217.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:21.341464 29544 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 696121417ea9403990456cfc6fc4f416 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:01:21.349236 29554 catalog_manager.cc:5484] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d reported cstate change: config changed from index 3 to 4, VOTER 4c8da1d580614449a636b855bf11b95f (127.27.217.193) evicted. New cstate: current_term: 1 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:01:21.365239 29650 tablet_service.cc:1514] Processing DeleteTablet for tablet 696121417ea9403990456cfc6fc4f416 with delete_type TABLET_DATA_TOMBSTONED (TS 4c8da1d580614449a636b855bf11b95f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59124
I20240501 20:01:21.366959 30608 tablet_replica.cc:330] T 696121417ea9403990456cfc6fc4f416 P 4c8da1d580614449a636b855bf11b95f: stopping tablet replica
I20240501 20:01:21.367941 30608 raft_consensus.cc:2229] T 696121417ea9403990456cfc6fc4f416 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:21.368752 30608 raft_consensus.cc:2258] T 696121417ea9403990456cfc6fc4f416 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:21.375108 30608 ts_tablet_manager.cc:1875] T 696121417ea9403990456cfc6fc4f416 P 4c8da1d580614449a636b855bf11b95f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:01:21.390285 30608 ts_tablet_manager.cc:1888] T 696121417ea9403990456cfc6fc4f416 P 4c8da1d580614449a636b855bf11b95f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:01:21.390720 30608 log.cc:1196] T 696121417ea9403990456cfc6fc4f416 P 4c8da1d580614449a636b855bf11b95f: Deleting WAL directory at /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714593602638719-28519-0/minicluster-data/ts-0-root/wals/696121417ea9403990456cfc6fc4f416
I20240501 20:01:21.392215 29542 catalog_manager.cc:4830] TS 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): tablet 696121417ea9403990456cfc6fc4f416 (table test-workload [id=c4cd6efa2a67402dac287d449c66135b]) successfully deleted
I20240501 20:01:21.406932 30488 raft_consensus.cc:1050] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f: attempting to promote NON_VOTER 888d99522e4844118f8f1cf26f268c8c to VOTER
I20240501 20:01:21.408849 30488 consensus_queue.cc:237] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:21.413863 30420 raft_consensus.cc:1261] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [term 1 LEARNER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:01:21.413856 29809 raft_consensus.cc:1261] T 7ed27033ab1645e9ab0600ab635d9589 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:01:21.415624 29730 raft_consensus.cc:1261] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:01:21.415843 30553 consensus_queue.cc:1035] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:21.417114 30551 consensus_queue.cc:1035] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:21.419255 30488 consensus_queue.cc:1035] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:21.443080 30488 raft_consensus.cc:2932] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:21.444679 29809 raft_consensus.cc:2932] T 7ed27033ab1645e9ab0600ab635d9589 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:21.454466 29730 raft_consensus.cc:2932] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:21.455302 30420 raft_consensus.cc:2932] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:21.461553 29556 catalog_manager.cc:5484] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f reported cstate change: config changed from index 2 to 3, 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:21.497354 29653 consensus_queue.cc:237] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:21.501545 30420 raft_consensus.cc:1261] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:01:21.502892 29730 raft_consensus.cc:1261] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:01:21.503639 30488 consensus_queue.cc:1035] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:01:21.504549 30553 consensus_queue.cc:1035] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240501 20:01:21.513823 30488 raft_consensus.cc:2932] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:21.515589 30420 raft_consensus.cc:2932] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:21.521746 29542 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7ed27033ab1645e9ab0600ab635d9589 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:01:21.529505 29556 catalog_manager.cc:5484] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f reported cstate change: config changed from index 3 to 4, VOTER ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) evicted. New cstate: current_term: 1 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:01:21.521001 29730 raft_consensus.cc:2932] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:21.544572 29793 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ed27033ab1645e9ab0600ab635d9589 with delete_type TABLET_DATA_TOMBSTONED (TS ee0d97247d394a8facfa774f15f3a67d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41856
I20240501 20:01:21.545574 30597 tablet_replica.cc:330] T 7ed27033ab1645e9ab0600ab635d9589 P ee0d97247d394a8facfa774f15f3a67d: stopping tablet replica
I20240501 20:01:21.546547 30597 raft_consensus.cc:2229] T 7ed27033ab1645e9ab0600ab635d9589 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:21.547153 30597 raft_consensus.cc:2258] T 7ed27033ab1645e9ab0600ab635d9589 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:21.550272 30597 ts_tablet_manager.cc:1875] T 7ed27033ab1645e9ab0600ab635d9589 P ee0d97247d394a8facfa774f15f3a67d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:01:21.569589 30597 ts_tablet_manager.cc:1888] T 7ed27033ab1645e9ab0600ab635d9589 P ee0d97247d394a8facfa774f15f3a67d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:01:21.569898 30597 log.cc:1196] T 7ed27033ab1645e9ab0600ab635d9589 P ee0d97247d394a8facfa774f15f3a67d: Deleting WAL directory at /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714593602638719-28519-0/minicluster-data/ts-2-root/wals/7ed27033ab1645e9ab0600ab635d9589
I20240501 20:01:21.571043 29544 catalog_manager.cc:4830] TS ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861): tablet 7ed27033ab1645e9ab0600ab635d9589 (table test-workload [id=c4cd6efa2a67402dac287d449c66135b]) successfully deleted
I20240501 20:01:22.595008 30626 raft_consensus.cc:482] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:22.595803 30626 raft_consensus.cc:504] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:22.598347 30626 leader_election.cc:288] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:22.709801 30626 raft_consensus.cc:482] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:22.710602 30626 raft_consensus.cc:504] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:22.713672 30625 raft_consensus.cc:482] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:22.714392 30625 raft_consensus.cc:504] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:22.716827 30624 raft_consensus.cc:482] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:22.717489 30624 raft_consensus.cc:504] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:22.757408 30626 leader_election.cc:288] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:22.719985 30624 leader_election.cc:288] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:22.761907 30625 leader_election.cc:288] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:22.828970 30624 raft_consensus.cc:482] T 3c5345f6eb62449ba69b4a9c2e349b31 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:22.829861 30624 raft_consensus.cc:504] T 3c5345f6eb62449ba69b4a9c2e349b31 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:22.832549 30624 leader_election.cc:288] T 3c5345f6eb62449ba69b4a9c2e349b31 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:22.843075 30624 raft_consensus.cc:482] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:22.983719 30623 raft_consensus.cc:482] T 010ceabf3e5a4b1c9330b80dade765bb P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:23.139787 30623 raft_consensus.cc:504] T 010ceabf3e5a4b1c9330b80dade765bb P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:23.142658 30623 leader_election.cc:288] T 010ceabf3e5a4b1c9330b80dade765bb P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:23.143322 30623 raft_consensus.cc:482] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:23.143900 30623 raft_consensus.cc:504] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:23.146793 30623 leader_election.cc:288] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:23.058915 30625 raft_consensus.cc:482] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:23.147748 30625 raft_consensus.cc:504] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:23.150379 30625 leader_election.cc:288] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:23.111447 30626 raft_consensus.cc:482] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:23.151180 30626 raft_consensus.cc:504] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:23.153738 30626 leader_election.cc:288] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:23.161065 30624 raft_consensus.cc:504] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:23.163851 30624 leader_election.cc:288] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:23.376428 30626 raft_consensus.cc:482] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:23.377264 30626 raft_consensus.cc:504] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:23.379729 30626 leader_election.cc:288] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:23.392239 29734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 550) took 1467 ms. Trace:
I20240501 20:01:23.392738 29734 rpcz_store.cc:276] 0501 20:01:21.924410 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:23.391189 (+1466779us) service_pool.cc:225] Handling call
0501 20:01:23.391594 (+ 405us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:23.391781 (+ 187us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:23.391808 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:23.391853 (+ 45us) raft_consensus.cc:1631] Marking committed up to 1
0501 20:01:23.391885 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:23.391938 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:23.392073 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:23.393901 30625 raft_consensus.cc:482] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:23.394469 30625 raft_consensus.cc:504] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:23.397284 30625 leader_election.cc:288] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:23.399391 30623 raft_consensus.cc:482] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:23.399860 30623 raft_consensus.cc:504] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:23.400991 30627 raft_consensus.cc:482] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:23.401393 30627 raft_consensus.cc:504] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:23.401661 30623 leader_election.cc:288] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:23.402176 30623 raft_consensus.cc:482] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:23.402526 30623 raft_consensus.cc:504] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:23.403290 30627 leader_election.cc:288] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:23.403693 30627 raft_consensus.cc:482] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:23.403873 30623 leader_election.cc:288] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:23.404006 30627 raft_consensus.cc:504] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:23.404366 30623 raft_consensus.cc:482] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:23.404844 30623 raft_consensus.cc:504] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:23.405537 30627 leader_election.cc:288] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:23.406432 29698 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.572s user 0.003s sys 0.000s
I20240501 20:01:23.406659 30623 leader_election.cc:288] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:23.407161 30623 raft_consensus.cc:482] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
W20240501 20:01:23.406778 29698 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.572s user 0.003s sys 0.000s
I20240501 20:01:23.408548 30624 raft_consensus.cc:482] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:23.409035 30624 raft_consensus.cc:504] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:23.410915 30624 leader_election.cc:288] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:23.407562 30623 raft_consensus.cc:504] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:23.412849 30623 leader_election.cc:288] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:23.413841 30627 raft_consensus.cc:482] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:23.414328 30627 raft_consensus.cc:504] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:23.416183 30627 leader_election.cc:288] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:23.416307 30625 raft_consensus.cc:482] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:23.416822 30625 raft_consensus.cc:504] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:23.418653 30625 leader_election.cc:288] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:23.419121 30625 raft_consensus.cc:482] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:23.416183 30626 raft_consensus.cc:482] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:23.420573 30626 raft_consensus.cc:504] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:23.422722 30627 raft_consensus.cc:482] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:23.419544 30625 raft_consensus.cc:504] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:23.423214 30627 raft_consensus.cc:504] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:23.425297 30626 leader_election.cc:288] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:23.431043 30631 raft_consensus.cc:482] T 1ca3948261374cef8b1b206a43c00173 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:23.426220 30627 leader_election.cc:288] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:23.426340 30625 leader_election.cc:288] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:23.426600 30629 raft_consensus.cc:482] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:23.431607 30631 raft_consensus.cc:504] T 1ca3948261374cef8b1b206a43c00173 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:23.438040 30627 raft_consensus.cc:482] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:23.438289 29729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 551) took 1513 ms. Trace:
I20240501 20:01:23.438616 30627 raft_consensus.cc:504] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:23.439013 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f5d1686214485d91a0baf4b870740e" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:23.439635 29653 raft_consensus.cc:2454] T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
I20240501 20:01:23.440577 30627 leader_election.cc:288] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:23.440959 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8787bc4d6b44649b9138d906730110a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:23.443001 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e49dd5180f04313ab33942fb3ddc147" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:23.444327 29652 raft_consensus.cc:2454] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
I20240501 20:01:23.444257 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:23.450479 29656 raft_consensus.cc:2454] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
I20240501 20:01:23.445595 30627 raft_consensus.cc:482] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:23.455786 29728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 552) took 1105 ms. Trace:
I20240501 20:01:23.455518 30627 raft_consensus.cc:504] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:23.438347 30631 leader_election.cc:288] T 1ca3948261374cef8b1b206a43c00173 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:23.456308 29728 rpcz_store.cc:276] 0501 20:01:22.350008 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:23.454841 (+1104833us) service_pool.cc:225] Handling call
0501 20:01:23.455158 (+ 317us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:23.455325 (+ 167us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:23.455355 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:23.455402 (+ 47us) raft_consensus.cc:1631] Marking committed up to 4
0501 20:01:23.455442 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:23.455509 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:23.455630 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:23.463196 29726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 548) took 1539 ms. Trace:
I20240501 20:01:23.464203 29726 rpcz_store.cc:276] 0501 20:01:21.923667 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:23.462427 (+1538760us) service_pool.cc:225] Handling call
0501 20:01:23.462720 (+ 293us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:23.462890 (+ 170us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:23.462916 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:23.462957 (+ 41us) raft_consensus.cc:1631] Marking committed up to 1
0501 20:01:23.462985 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:23.463039 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:23.463139 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:23.472872 30631 raft_consensus.cc:482] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:23.476809 30617 raft_consensus.cc:482] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:23.477427 30617 raft_consensus.cc:504] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:23.479564 30617 leader_election.cc:288] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:23.438875 29729 rpcz_store.cc:276] 0501 20:01:21.924726 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:23.437034 (+1512308us) service_pool.cc:225] Handling call
0501 20:01:23.437445 (+ 411us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:23.437672 (+ 227us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:23.437744 (+ 72us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:23.437803 (+ 59us) raft_consensus.cc:1631] Marking committed up to 1
0501 20:01:23.437855 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:23.437936 (+ 81us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:23.438091 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:23.480623 30633 raft_consensus.cc:482] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:23.481200 30633 raft_consensus.cc:504] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:23.483355 30633 leader_election.cc:288] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:23.483927 30632 raft_consensus.cc:482] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
W20240501 20:01:23.493894 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 48626.6us
I20240501 20:01:23.484506 30632 raft_consensus.cc:504] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:23.495755 30627 leader_election.cc:288] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:23.475176 30631 raft_consensus.cc:504] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:23.497064 30632 leader_election.cc:288] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:23.486685 30586 raft_consensus.cc:482] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:23.498396 30631 leader_election.cc:288] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:23.498618 30617 raft_consensus.cc:482] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:23.498432 29732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 547) took 1780 ms. Trace:
I20240501 20:01:23.499783 29732 rpcz_store.cc:276] 0501 20:01:21.718076 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:23.497631 (+1779555us) service_pool.cc:225] Handling call
0501 20:01:23.497941 (+ 310us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:23.498111 (+ 170us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:23.498136 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:23.498176 (+ 40us) raft_consensus.cc:1631] Marking committed up to 1
0501 20:01:23.498204 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:23.498254 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:23.498376 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:23.509508 30639 raft_consensus.cc:482] T 456035a1cb3f4f7da52d0fc1dd1c656c P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:23.499326 30617 raft_consensus.cc:504] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:23.514284 29698 leader_election.cc:302] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:23.510453 30639 raft_consensus.cc:504] T 456035a1cb3f4f7da52d0fc1dd1c656c P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:23.521533 30617 leader_election.cc:288] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:23.522177 30617 raft_consensus.cc:482] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:23.522702 30632 raft_consensus.cc:482] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:23.524351 30632 raft_consensus.cc:504] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:23.526217 30632 leader_election.cc:288] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:23.528729 30637 raft_consensus.cc:2781] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:23.529242 30637 raft_consensus.cc:482] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:23.529703 30637 raft_consensus.cc:3037] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:23.498423 30586 raft_consensus.cc:504] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } }
I20240501 20:01:23.432929 30629 raft_consensus.cc:504] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:23.538869 30639 leader_election.cc:288] T 456035a1cb3f4f7da52d0fc1dd1c656c P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:23.552814 29698 leader_election.cc:302] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:23.522704 30617 raft_consensus.cc:504] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:23.552997 30627 raft_consensus.cc:482] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:23.575012 30627 raft_consensus.cc:504] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:23.577567 30627 leader_election.cc:288] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:23.622509 29731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 546) took 1904 ms. Trace:
I20240501 20:01:23.623127 29731 rpcz_store.cc:276] 0501 20:01:21.717763 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:23.621407 (+1903644us) service_pool.cc:225] Handling call
0501 20:01:23.621860 (+ 453us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:23.622090 (+ 230us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:23.622127 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:23.622179 (+ 52us) raft_consensus.cc:1631] Marking committed up to 1
0501 20:01:23.622222 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:23.622289 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:23.622438 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:23.631023 30640 raft_consensus.cc:482] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:23.632017 30641 raft_consensus.cc:2781] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:23.631825 30640 raft_consensus.cc:504] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:23.637989 30630 raft_consensus.cc:482] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
W20240501 20:01:23.630262 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 77533us
I20240501 20:01:23.636035 30633 raft_consensus.cc:482] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:23.642176 30633 raft_consensus.cc:504] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:23.642832 30635 raft_consensus.cc:482] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:23.628968 30617 leader_election.cc:288] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:23.652869 29698 leader_election.cc:302] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:23.657583 30629 leader_election.cc:288] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:23.658350 30629 raft_consensus.cc:2781] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:23.658784 30629 raft_consensus.cc:482] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:23.659216 30629 raft_consensus.cc:3037] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:23.638679 30630 raft_consensus.cc:504] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:23.637317 30586 leader_election.cc:288] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:23.643512 30635 raft_consensus.cc:504] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:23.632647 30641 raft_consensus.cc:482] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:23.640765 30632 raft_consensus.cc:482] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:23.644196 30633 leader_election.cc:288] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:23.648273 30631 raft_consensus.cc:482] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:23.763296 30641 raft_consensus.cc:3037] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:23.699941 30586 raft_consensus.cc:482] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:24.081722 30639 raft_consensus.cc:482] T 3c5345f6eb62449ba69b4a9c2e349b31 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:24.082170 30630 leader_election.cc:288] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:24.082336 30617 raft_consensus.cc:482] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:24.082789 30630 raft_consensus.cc:482] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:24.082912 30617 raft_consensus.cc:504] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.083313 30630 raft_consensus.cc:504] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:24.084796 30635 leader_election.cc:288] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:24.085304 30617 leader_election.cc:288] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:24.081768 30497 raft_consensus.cc:482] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:24.086114 30497 raft_consensus.cc:504] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.088318 30497 leader_election.cc:288] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:24.092677 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49da4246f5f6438abe0e7e8c9a523d80" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:24.093636 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649b71f9f267437ca51c758dd9ca6987" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:24.094565 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211e27a3b0084b89a6dd7550849b5a65" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:24.097368 30586 raft_consensus.cc:504] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.099953 29800 raft_consensus.cc:2454] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:23.771521 30640 leader_election.cc:288] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:24.082446 30639 raft_consensus.cc:504] T 3c5345f6eb62449ba69b4a9c2e349b31 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.078640 30631 raft_consensus.cc:504] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.109009 30631 leader_election.cc:288] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:24.056450 30632 raft_consensus.cc:504] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.112449 30632 leader_election.cc:288] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:24.114552 30639 leader_election.cc:288] T 3c5345f6eb62449ba69b4a9c2e349b31 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:24.116753 30630 leader_election.cc:288] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:24.099882 30586 leader_election.cc:288] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
W20240501 20:01:24.124272 30637 consensus_meta.cc:287] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d: Time spent flushing consensus metadata: real 0.594s user 0.004s sys 0.005s
I20240501 20:01:24.125409 30637 raft_consensus.cc:504] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.128036 30637 leader_election.cc:288] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:24.128585 30629 raft_consensus.cc:504] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.132699 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8edcd903224d48f889b46e1b5318e0ec" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:24.133270 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a2bba62f434d1c8489b55c8fec2187" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:24.133561 29805 raft_consensus.cc:2454] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:24.133889 29808 raft_consensus.cc:2454] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:24.134721 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31fff64b453a423eb0f567b3171564fb" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:24.134917 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07bdb8b09b8c4ef4b8e49daa9b8c01d6" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:24.135277 29808 raft_consensus.cc:2454] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:24.136116 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece8d4ed84c34b748f7b6272b31c90ac" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:24.136490 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:24.136862 29808 raft_consensus.cc:2454] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:24.137868 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a35893b91247a99912eba787d787cf" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:24.138553 29808 raft_consensus.cc:2454] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:24.138592 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcf30cf66714efdac997862dafbe4a2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:24.135741 29805 raft_consensus.cc:2454] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:24.139616 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458740ab5c444b6ab3d1991c69db5084" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:24.140731 29808 raft_consensus.cc:2454] T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:24.140945 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1a08b455b94ae089396a15c837ddc2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:24.141634 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f261d2a05bf4c838532049601694b4a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:24.146999 29804 raft_consensus.cc:2454] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:24.147972 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5525f546426c47a38edf7f56ad04b7ab" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:24.142205 30629 leader_election.cc:288] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:24.131599 30641 raft_consensus.cc:504] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.152138 30641 leader_election.cc:288] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:24.169013 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0374bb790ed345ed90452595f71383a2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:24.169936 29808 raft_consensus.cc:2454] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:24.172009 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c651044a305c4168805825a53d910899" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:24.243198 29626 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.702s user 0.014s sys 0.009s
W20240501 20:01:24.244127 29626 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.704s user 0.014s sys 0.010s
I20240501 20:01:24.281129 29802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010ceabf3e5a4b1c9330b80dade765bb" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:24.289114 29756 debug-util.cc:398] Leaking SignalData structure 0x7b0800174c80 after lost signal to thread 28522
I20240501 20:01:24.282779 29727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 554) took 1844 ms. Trace:
W20240501 20:01:24.285514 29698 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.630s user 0.043s sys 0.051s
W20240501 20:01:24.293731 29698 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.638s user 0.043s sys 0.051s
W20240501 20:01:24.294297 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 638663us
I20240501 20:01:24.296464 30637 raft_consensus.cc:482] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:24.297902 30637 raft_consensus.cc:504] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:24.309484 29699 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.664s user 0.002s sys 0.008s
I20240501 20:01:24.308971 30637 leader_election.cc:288] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:24.311803 29699 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.671s user 0.002s sys 0.008s
I20240501 20:01:24.312858 30648 raft_consensus.cc:482] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:24.306545 29727 rpcz_store.cc:276] 0501 20:01:22.438363 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:24.280278 (+1841915us) service_pool.cc:225] Handling call
0501 20:01:24.281285 (+ 1007us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:24.281893 (+ 608us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:24.281961 (+ 68us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:24.282055 (+ 94us) raft_consensus.cc:1631] Marking committed up to 1
0501 20:01:24.282120 (+ 65us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:24.282252 (+ 132us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:24.282464 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:24.320963 30648 raft_consensus.cc:504] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:24.324012 29625 leader_election.cc:302] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:24.326145 30648 leader_election.cc:288] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:24.326761 30648 raft_consensus.cc:482] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:24.327742 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd5e83faaff4e2da4e4cc22ba410d33" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:24.327603 30648 raft_consensus.cc:504] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:24.331537 30648 leader_election.cc:288] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:24.332376 30648 raft_consensus.cc:482] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:24.333274 30648 raft_consensus.cc:504] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:24.336352 30648 leader_election.cc:288] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:24.337549 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e89c0572e4ee7be8e5315b459bdb3" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:24.341943 30629 raft_consensus.cc:482] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:24.342907 30629 raft_consensus.cc:504] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } }
I20240501 20:01:24.343688 30648 raft_consensus.cc:482] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:24.344380 30630 raft_consensus.cc:482] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:24.346372 30632 raft_consensus.cc:482] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:24.347357 30632 raft_consensus.cc:504] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.347971 30637 raft_consensus.cc:482] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:24.348815 30637 raft_consensus.cc:504] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.349910 30632 leader_election.cc:288] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:24.351061 30637 leader_election.cc:288] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:24.352991 30640 raft_consensus.cc:482] T c651044a305c4168805825a53d910899 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:24.353739 30640 raft_consensus.cc:504] T c651044a305c4168805825a53d910899 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.338552 30650 raft_consensus.cc:482] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:24.363894 30633 raft_consensus.cc:482] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:24.351616 30637 raft_consensus.cc:482] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:24.357111 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5345f6eb62449ba69b4a9c2e349b31" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:24.344657 30648 raft_consensus.cc:504] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:24.358469 30640 leader_election.cc:288] T c651044a305c4168805825a53d910899 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:24.374588 30629 leader_election.cc:288] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861), 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959)
I20240501 20:01:24.374385 29807 raft_consensus.cc:2454] T 3c5345f6eb62449ba69b4a9c2e349b31 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:24.364605 30631 raft_consensus.cc:482] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:24.375451 30633 raft_consensus.cc:504] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.376185 30631 raft_consensus.cc:504] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.378302 30633 leader_election.cc:288] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:24.365164 30630 raft_consensus.cc:504] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.378558 30631 leader_election.cc:288] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:24.378492 30644 raft_consensus.cc:482] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:24.380589 30644 raft_consensus.cc:504] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.374308 30636 raft_consensus.cc:482] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:24.372659 30646 raft_consensus.cc:482] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:24.382874 30636 raft_consensus.cc:504] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.384636 30651 raft_consensus.cc:482] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:24.385102 30644 leader_election.cc:288] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:24.386102 29625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 62356.3us
I20240501 20:01:24.387449 30637 raft_consensus.cc:504] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.388152 29625 leader_election.cc:302] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:24.386528 30630 leader_election.cc:288] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:24.392000 30646 raft_consensus.cc:504] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.398002 30643 raft_consensus.cc:2781] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:24.398484 30643 raft_consensus.cc:482] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:24.398824 30636 leader_election.cc:288] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:24.398942 30643 raft_consensus.cc:3037] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:24.401017 29625 leader_election.cc:302] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:24.407613 30654 raft_consensus.cc:482] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:24.408792 30637 leader_election.cc:288] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:24.407816 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd5e83faaff4e2da4e4cc22ba410d33" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:24.386646 30650 raft_consensus.cc:504] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.418721 29727 raft_consensus.cc:2454] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 888d99522e4844118f8f1cf26f268c8c in term 3.
I20240501 20:01:24.420362 30615 raft_consensus.cc:482] T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:24.421344 30615 raft_consensus.cc:504] T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.400249 30649 raft_consensus.cc:482] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:24.430030 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e38a2e91a3d48d09a22abce723af833" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:24.432528 29806 raft_consensus.cc:2454] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:24.437477 30619 raft_consensus.cc:482] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:24.425901 29625 leader_election.cc:302] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:24.441781 30618 raft_consensus.cc:482] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:24.441920 30616 raft_consensus.cc:482] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:24.442021 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9046e38fc57f4364aa3d539e1fc01928" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:24.442404 30419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd5e83faaff4e2da4e4cc22ba410d33" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c" is_pre_election: true
I20240501 20:01:24.442526 30618 raft_consensus.cc:504] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.445852 30586 raft_consensus.cc:482] T 84fa28fb55f44b61ba1a88ed775a78ba P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:24.408811 30654 raft_consensus.cc:504] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.444182 30419 raft_consensus.cc:2454] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 3.
I20240501 20:01:24.446681 30586 raft_consensus.cc:504] T 84fa28fb55f44b61ba1a88ed775a78ba P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.449971 30615 leader_election.cc:288] T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
W20240501 20:01:24.445138 29756 debug-util.cc:398] Leaking SignalData structure 0x7b08000cdfc0 after lost signal to thread 29592
I20240501 20:01:24.450565 30652 raft_consensus.cc:482] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:24.451010 30620 raft_consensus.cc:482] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:24.451138 30648 leader_election.cc:288] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:24.438184 30619 raft_consensus.cc:504] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.457698 30624 raft_consensus.cc:2781] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:24.415572 30651 raft_consensus.cc:504] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:24.442730 30616 raft_consensus.cc:504] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.472867 30639 raft_consensus.cc:482] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:24.473670 30639 raft_consensus.cc:504] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.476275 30653 raft_consensus.cc:482] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:24.468263 30646 leader_election.cc:288] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:24.468555 30624 raft_consensus.cc:482] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:24.470053 30648 raft_consensus.cc:482] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:24.488865 30624 raft_consensus.cc:3037] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:24.496634 30658 raft_consensus.cc:482] T add9d7579351483fb268f89a44e95d8a P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:24.474428 30615 raft_consensus.cc:482] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:24.496718 30655 raft_consensus.cc:2781] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:24.499243 30586 leader_election.cc:288] T 84fa28fb55f44b61ba1a88ed775a78ba P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:24.508510 30613 raft_consensus.cc:482] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:24.491072 30650 leader_election.cc:288] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:24.514266 29625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 106509us
I20240501 20:01:24.490351 30620 raft_consensus.cc:504] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.516798 29625 leader_election.cc:302] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:24.519804 30642 raft_consensus.cc:482] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:24.508893 30648 raft_consensus.cc:504] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:24.476114 30652 raft_consensus.cc:504] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:24.497471 30658 raft_consensus.cc:504] T add9d7579351483fb268f89a44e95d8a P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:24.529274 30419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ed27033ab1645e9ab0600ab635d9589" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c" is_pre_election: true
I20240501 20:01:24.530611 30617 raft_consensus.cc:482] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:24.531076 29625 leader_election.cc:302] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:24.528362 30614 raft_consensus.cc:482] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:24.528191 30650 raft_consensus.cc:482] T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:24.531680 30648 leader_election.cc:288] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:24.514449 30646 raft_consensus.cc:2781] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:24.540202 30652 leader_election.cc:288] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:24.537094 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a2bba62f434d1c8489b55c8fec2187" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:24.541404 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e07220db7444ae97f294d70eda6a93" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:24.544858 30616 leader_election.cc:288] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:24.546865 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ed27033ab1645e9ab0600ab635d9589" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:24.547917 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add9d7579351483fb268f89a44e95d8a" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:24.513146 30655 raft_consensus.cc:482] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:24.537551 30651 leader_election.cc:288] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:24.551090 30646 raft_consensus.cc:482] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:24.552814 30619 leader_election.cc:288] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:24.520889 30642 raft_consensus.cc:504] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.522058 30618 leader_election.cc:288] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:24.552964 30639 leader_election.cc:288] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:24.487218 30649 raft_consensus.cc:504] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.557014 30654 leader_election.cc:288] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:24.549641 30419 raft_consensus.cc:2454] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
I20240501 20:01:24.559039 30646 raft_consensus.cc:3037] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:24.535342 30615 raft_consensus.cc:504] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.557663 30655 raft_consensus.cc:3037] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:24.520076 30613 raft_consensus.cc:504] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.554338 30658 leader_election.cc:288] T add9d7579351483fb268f89a44e95d8a P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:24.489810 30653 raft_consensus.cc:504] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.575408 29653 raft_consensus.cc:2454] T add9d7579351483fb268f89a44e95d8a P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 888d99522e4844118f8f1cf26f268c8c in term 1.
I20240501 20:01:24.575634 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c597f15e844f30837012baed4b0501" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:24.577060 30617 raft_consensus.cc:504] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
W20240501 20:01:24.574898 29756 debug-util.cc:398] Leaking SignalData structure 0x7b08003101e0 after lost signal to thread 29684
I20240501 20:01:24.553634 30614 raft_consensus.cc:504] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.582485 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "456035a1cb3f4f7da52d0fc1dd1c656c" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:24.583225 29731 raft_consensus.cc:2454] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
I20240501 20:01:24.589555 30653 leader_election.cc:288] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:24.575958 30618 raft_consensus.cc:482] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:24.590728 30618 raft_consensus.cc:504] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:24.591131 30615 leader_election.cc:288] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:24.591363 30419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "696121417ea9403990456cfc6fc4f416" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c" is_pre_election: true
I20240501 20:01:24.573889 30639 raft_consensus.cc:482] T ec073fe645614fe28e7837de3c22510b P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:24.592955 29732 raft_consensus.cc:2454] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
I20240501 20:01:24.594880 30419 raft_consensus.cc:2454] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
I20240501 20:01:24.578656 30383 leader_election.cc:302] T add9d7579351483fb268f89a44e95d8a P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, 888d99522e4844118f8f1cf26f268c8c; no voters:
I20240501 20:01:24.582196 30620 leader_election.cc:288] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:24.595713 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5525fc4b82a44b0991c3dd6098fde81" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:24.605567 30617 leader_election.cc:288] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:24.606398 30639 raft_consensus.cc:504] T ec073fe645614fe28e7837de3c22510b P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.612937 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69915dd5992641e3a24f7f45f1cd5c0c" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:24.613986 30657 raft_consensus.cc:482] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:24.614210 29731 raft_consensus.cc:2454] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
I20240501 20:01:24.580192 30650 raft_consensus.cc:504] T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.616355 30420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e07220db7444ae97f294d70eda6a93" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c" is_pre_election: true
I20240501 20:01:24.596657 30654 raft_consensus.cc:2781] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:24.616490 30657 raft_consensus.cc:504] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.620963 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e07220db7444ae97f294d70eda6a93" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:24.623095 29731 raft_consensus.cc:2454] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 888d99522e4844118f8f1cf26f268c8c in term 1.
I20240501 20:01:24.621491 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05df43bdfa479b902b1da297fb5974" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:24.620345 30420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05df43bdfa479b902b1da297fb5974" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c" is_pre_election: true
I20240501 20:01:24.620898 30654 raft_consensus.cc:482] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:24.625241 30641 raft_consensus.cc:482] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:24.625399 30654 raft_consensus.cc:3037] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:24.625584 30642 leader_election.cc:288] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:24.625893 30641 raft_consensus.cc:504] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.628602 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "696121417ea9403990456cfc6fc4f416" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:24.629240 30639 leader_election.cc:288] T ec073fe645614fe28e7837de3c22510b P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:24.629351 30658 raft_consensus.cc:2781] T add9d7579351483fb268f89a44e95d8a P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:24.629801 30658 raft_consensus.cc:482] T add9d7579351483fb268f89a44e95d8a P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:24.631641 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ed27033ab1645e9ab0600ab635d9589" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:24.613269 30653 raft_consensus.cc:482] T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:24.618702 30660 raft_consensus.cc:2781] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:24.632309 30618 leader_election.cc:288] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:24.594547 30643 raft_consensus.cc:504] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.623320 29727 raft_consensus.cc:2454] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
I20240501 20:01:24.644613 30384 leader_election.cc:302] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 888d99522e4844118f8f1cf26f268c8c, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:24.644958 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9046e38fc57f4364aa3d539e1fc01928" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:24.638242 30643 leader_election.cc:288] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:24.638334 30655 raft_consensus.cc:504] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.638513 30641 leader_election.cc:288] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:24.646394 30384 leader_election.cc:302] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 888d99522e4844118f8f1cf26f268c8c, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:24.647321 30652 raft_consensus.cc:2781] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:24.647874 30652 raft_consensus.cc:482] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:24.636648 30613 leader_election.cc:288] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:24.648284 30651 raft_consensus.cc:2781] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:24.648535 30652 raft_consensus.cc:3037] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:24.632458 29729 raft_consensus.cc:2454] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 888d99522e4844118f8f1cf26f268c8c in term 1.
I20240501 20:01:24.647262 30420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e89c0572e4ee7be8e5315b459bdb3" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c" is_pre_election: true
I20240501 20:01:24.658437 30420 raft_consensus.cc:2454] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
I20240501 20:01:24.660284 29801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "696121417ea9403990456cfc6fc4f416" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:24.660410 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05df43bdfa479b902b1da297fb5974" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:24.661118 29806 raft_consensus.cc:2454] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 888d99522e4844118f8f1cf26f268c8c in term 1.
I20240501 20:01:24.635785 30645 raft_consensus.cc:482] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:24.630235 30658 raft_consensus.cc:3037] T add9d7579351483fb268f89a44e95d8a P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:24.661208 30649 leader_election.cc:288] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:24.656540 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e89c0572e4ee7be8e5315b459bdb3" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:24.655320 30655 leader_election.cc:288] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:24.665741 29803 raft_consensus.cc:2454] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 888d99522e4844118f8f1cf26f268c8c in term 1.
W20240501 20:01:24.666759 29625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 135813us
W20240501 20:01:24.667465 29756 debug-util.cc:398] Leaking SignalData structure 0x7b080031aac0 after lost signal to thread 29759
I20240501 20:01:24.667575 29625 leader_election.cc:302] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:24.634868 30650 leader_election.cc:288] T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:24.630863 30383 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 49373.5us
I20240501 20:01:24.635392 30660 raft_consensus.cc:482] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:24.672490 30660 raft_consensus.cc:3037] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:24.670630 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:24.671526 30655 raft_consensus.cc:2781] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:24.671634 30657 leader_election.cc:288] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:24.655902 30614 leader_election.cc:288] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:24.658197 30651 raft_consensus.cc:482] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:24.662592 30645 raft_consensus.cc:504] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.635162 30653 raft_consensus.cc:504] T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.677781 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add9d7579351483fb268f89a44e95d8a" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:24.675264 30655 raft_consensus.cc:482] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:24.678421 30651 raft_consensus.cc:3037] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:24.684604 30655 raft_consensus.cc:3037] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:24.637989 30624 raft_consensus.cc:504] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.690441 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48fd0a8e289401aa7417c42ff2c81b2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:24.700621 29729 raft_consensus.cc:2454] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 3.
I20240501 20:01:24.677062 29727 raft_consensus.cc:2454] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
I20240501 20:01:24.715508 29772 leader_election.cc:302] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 888d99522e4844118f8f1cf26f268c8c, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:24.677760 30385 leader_election.cc:302] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 888d99522e4844118f8f1cf26f268c8c; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:24.718935 30663 raft_consensus.cc:2726] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:24.675541 30384 leader_election.cc:302] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 888d99522e4844118f8f1cf26f268c8c, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
W20240501 20:01:24.722333 30384 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 52178.9us
I20240501 20:01:24.725733 30663 raft_consensus.cc:2781] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:24.726294 30663 raft_consensus.cc:482] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:24.726712 30663 raft_consensus.cc:3037] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:01:24.728669 29772 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.196:37227, user_credentials={real_user=slave}} blocked reactor thread for 69232.4us
I20240501 20:01:24.729350 30614 raft_consensus.cc:2781] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:24.729391 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649b71f9f267437ca51c758dd9ca6987" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:24.729921 30614 raft_consensus.cc:482] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:24.730340 30614 raft_consensus.cc:3037] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:01:24.732611 30385 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 56653.6us
W20240501 20:01:24.727733 29756 debug-util.cc:398] Leaking SignalData structure 0x7b0800160ee0 after lost signal to thread 29833
I20240501 20:01:24.733407 30385 leader_election.cc:302] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 888d99522e4844118f8f1cf26f268c8c; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:24.734848 30645 leader_election.cc:288] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:24.739233 29801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8787bc4d6b44649b9138d906730110a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:24.735630 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f5d1686214485d91a0baf4b870740e" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:24.740159 30648 raft_consensus.cc:2726] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:24.738830 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e49dd5180f04313ab33942fb3ddc147" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:24.740207 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:24.743515 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca3948261374cef8b1b206a43c00173" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:24.740247 29756 debug-util.cc:398] Leaking SignalData structure 0x7b0800281480 after lost signal to thread 30445
I20240501 20:01:24.746318 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb27193abbf4644a48662909bc7b04a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:24.749265 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5525f546426c47a38edf7f56ad04b7ab" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:24.749461 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5345f6eb62449ba69b4a9c2e349b31" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:24.742012 30653 leader_election.cc:288] T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:24.741832 30649 raft_consensus.cc:482] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:24.748407 30624 leader_election.cc:288] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:24.751693 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec073fe645614fe28e7837de3c22510b" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:24.752357 29800 raft_consensus.cc:2454] T ec073fe645614fe28e7837de3c22510b P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
I20240501 20:01:24.753229 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1a08b455b94ae089396a15c837ddc2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:24.753036 30649 raft_consensus.cc:504] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.753150 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d2cac25f354a97b1bfb53d57ae7c08" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:24.755396 30649 leader_election.cc:288] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:24.760924 30385 leader_election.cc:302] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 888d99522e4844118f8f1cf26f268c8c, ee0d97247d394a8facfa774f15f3a67d; no voters: fdfeac4190e045f290e003b2298d988d
I20240501 20:01:24.761855 30648 raft_consensus.cc:2781] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:24.761139 29625 leader_election.cc:302] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:24.762274 30648 raft_consensus.cc:482] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:24.762878 30648 raft_consensus.cc:3037] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:24.763424 30653 raft_consensus.cc:2781] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:24.763880 30653 raft_consensus.cc:482] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:24.764832 30653 raft_consensus.cc:3037] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:24.764782 30385 leader_election.cc:302] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 888d99522e4844118f8f1cf26f268c8c, ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:24.769045 29727 raft_consensus.cc:1226] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:24.760270 30654 raft_consensus.cc:504] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.773118 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649b71f9f267437ca51c758dd9ca6987" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:24.774266 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49da4246f5f6438abe0e7e8c9a523d80" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:24.776316 29732 raft_consensus.cc:2454] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:24.774734 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e38a2e91a3d48d09a22abce723af833" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:24.777240 30654 leader_election.cc:288] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:24.777367 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5345f6eb62449ba69b4a9c2e349b31" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:24.778683 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8edcd903224d48f889b46e1b5318e0ec" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:24.779331 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010ceabf3e5a4b1c9330b80dade765bb" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:24.780779 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a2bba62f434d1c8489b55c8fec2187" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:24.781510 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:24.781592 29625 leader_election.cc:302] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:24.782215 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5525f546426c47a38edf7f56ad04b7ab" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:24.782545 30654 raft_consensus.cc:2781] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:24.782897 29728 raft_consensus.cc:2454] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:24.783028 30654 raft_consensus.cc:482] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:24.783528 30654 raft_consensus.cc:3037] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:24.785283 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211e27a3b0084b89a6dd7550849b5a65" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:24.787086 29625 leader_election.cc:302] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:24.793769 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211e27a3b0084b89a6dd7550849b5a65" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:24.794390 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be383be783234855878cb8fd442c7b05" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:24.801949 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "696121417ea9403990456cfc6fc4f416" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:24.802093 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:24.803370 29807 raft_consensus.cc:2454] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
I20240501 20:01:24.803139 30614 raft_consensus.cc:504] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:24.803938 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcf30cf66714efdac997862dafbe4a2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:24.796628 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c597f15e844f30837012baed4b0501" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:24.795316 29802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5525fc4b82a44b0991c3dd6098fde81" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:24.796993 29801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e49dd5180f04313ab33942fb3ddc147" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:24.796676 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f5d1686214485d91a0baf4b870740e" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:24.809693 30648 raft_consensus.cc:504] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:24.805282 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9472c53f79ec423f82c396bd980674e2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:24.794214 30655 raft_consensus.cc:504] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.802016 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48630ae6eb394ac1b54955dd01cf0ed2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:24.807983 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07bdb8b09b8c4ef4b8e49daa9b8c01d6" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:24.774744 30652 raft_consensus.cc:504] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:24.795310 30663 raft_consensus.cc:504] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:24.814414 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f261d2a05bf4c838532049601694b4a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:24.815017 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458740ab5c444b6ab3d1991c69db5084" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:24.838927 29625 leader_election.cc:302] T 3c5345f6eb62449ba69b4a9c2e349b31 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:24.814705 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31fff64b453a423eb0f567b3171564fb" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:24.839476 29653 consensus_queue.cc:237] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:24.835402 30624 raft_consensus.cc:2781] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:24.888362 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3f29f0eeea4b119984634202e5d9cd" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:24.882138 30614 leader_election.cc:288] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:24.852903 30667 raft_consensus.cc:2781] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:24.892774 30420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e89c0572e4ee7be8e5315b459bdb3" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c"
I20240501 20:01:24.892442 30665 raft_consensus.cc:2781] T 3c5345f6eb62449ba69b4a9c2e349b31 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:24.889326 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcf30cf66714efdac997862dafbe4a2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
W20240501 20:01:24.891577 29625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 70308.6us
I20240501 20:01:24.889464 30624 raft_consensus.cc:482] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:24.893610 29734 raft_consensus.cc:2454] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:24.806463 30658 raft_consensus.cc:504] T add9d7579351483fb268f89a44e95d8a P 888d99522e4844118f8f1cf26f268c8c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:24.836369 30660 raft_consensus.cc:504] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.890163 30663 leader_election.cc:288] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 2 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:24.839681 30651 raft_consensus.cc:504] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:24.899756 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8cd40cb4044d18a86001ce6ddf97ab" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:24.890377 30653 raft_consensus.cc:504] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.887460 30646 raft_consensus.cc:504] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.894110 30665 raft_consensus.cc:482] T 3c5345f6eb62449ba69b4a9c2e349b31 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:24.898939 30624 raft_consensus.cc:3037] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:24.897989 30667 raft_consensus.cc:482] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:24.898077 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece8d4ed84c34b748f7b6272b31c90ac" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:24.908736 30418 raft_consensus.cc:1226] T add9d7579351483fb268f89a44e95d8a P 888d99522e4844118f8f1cf26f268c8c [term 2 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:24.916306 30613 consensus_queue.cc:1046] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:01:24.917363 30613 raft_consensus.cc:3032] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:24.917778 30613 raft_consensus.cc:729] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:24.918668 30613 consensus_queue.cc:260] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:24.905483 29625 leader_election.cc:302] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:24.906704 30665 raft_consensus.cc:3037] T 3c5345f6eb62449ba69b4a9c2e349b31 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:24.911545 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69915dd5992641e3a24f7f45f1cd5c0c" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:24.912945 30653 leader_election.cc:288] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:24.920781 29654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ed27033ab1645e9ab0600ab635d9589" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:24.921312 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a35893b91247a99912eba787d787cf" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:24.910785 30667 raft_consensus.cc:3037] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:24.921862 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1a08b455b94ae089396a15c837ddc2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:24.927660 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b337b8e3e38d4ad6b1076fbd0d0a1609" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:24.922482 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e07220db7444ae97f294d70eda6a93" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:24.920804 30613 raft_consensus.cc:3037] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:24.922452 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd5e83faaff4e2da4e4cc22ba410d33" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:24.928866 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac2171683884f6f97cdfeb0a656cb49" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:24.922785 30652 leader_election.cc:288] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 4 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:24.902473 30648 leader_election.cc:288] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 2 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:24.924300 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05df43bdfa479b902b1da297fb5974" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:24.922092 30624 raft_consensus.cc:504] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:24.926529 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0374bb790ed345ed90452595f71383a2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:24.927300 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af08384057a4d85977481472f460168" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:24.930274 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84fa28fb55f44b61ba1a88ed775a78ba" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:24.919610 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77be7101c0fe4085b6706296f609870a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:24.948626 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d86f8848a184a87947fd927c44e5a35" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:24.924955 30651 leader_election.cc:288] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 2 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:24.920223 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c651044a305c4168805825a53d910899" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:24.929980 30660 leader_election.cc:288] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:24.950474 29625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 46508.6us
I20240501 20:01:24.956642 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "456035a1cb3f4f7da52d0fc1dd1c656c" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:24.986840 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49da4246f5f6438abe0e7e8c9a523d80" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:24.989154 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add9d7579351483fb268f89a44e95d8a" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:24.998399 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9472c53f79ec423f82c396bd980674e2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:24.983093 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec073fe645614fe28e7837de3c22510b" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:25.019857 30658 leader_election.cc:288] T add9d7579351483fb268f89a44e95d8a P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 2 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:24.986295 30646 leader_election.cc:288] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:25.021703 29809 raft_consensus.cc:2454] T 456035a1cb3f4f7da52d0fc1dd1c656c P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
I20240501 20:01:25.021922 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c597f15e844f30837012baed4b0501" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.057904 30655 leader_election.cc:288] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:24.983144 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38cbeb0dc0c14e3d9182101eea885625" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:24.992852 29802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c651044a305c4168805825a53d910899" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:25.052804 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668b4a4d782c4278932dd8413dc3ab3e" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:25.053021 30645 raft_consensus.cc:482] T 010ceabf3e5a4b1c9330b80dade765bb P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:25.058573 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69915dd5992641e3a24f7f45f1cd5c0c" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.028071 30653 raft_consensus.cc:2781] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:25.181056 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649b71f9f267437ca51c758dd9ca6987" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.183274 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05df43bdfa479b902b1da297fb5974" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:25.184403 29660 raft_consensus.cc:2379] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 2: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
W20240501 20:01:25.182781 29700 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.27.217.195:35861 timed out after 1.558s (SENT) [suppressed 1 similar messages]
I20240501 20:01:25.185099 29802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:25.127619 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af08384057a4d85977481472f460168" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.185365 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84fa28fb55f44b61ba1a88ed775a78ba" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.186735 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.187312 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9472c53f79ec423f82c396bd980674e2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.187443 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "456035a1cb3f4f7da52d0fc1dd1c656c" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.191148 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac2171683884f6f97cdfeb0a656cb49" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.069658 30624 leader_election.cc:288] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:25.191962 29800 raft_consensus.cc:2454] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 888d99522e4844118f8f1cf26f268c8c in term 2.
I20240501 20:01:25.192497 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36057b3f24d44b9aac3a1378053a395f" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:25.128836 29654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a2bba62f434d1c8489b55c8fec2187" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.193640 29657 raft_consensus.cc:2454] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
I20240501 20:01:25.193903 29654 raft_consensus.cc:2379] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 2: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:25.194574 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3f29f0eeea4b119984634202e5d9cd" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.195791 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e9f75387564275a60d38e9a910319c" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:25.195210 29657 raft_consensus.cc:2454] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
I20240501 20:01:25.196651 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfac893b4b4f457694a4eb184b6abb53" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.181885 30653 raft_consensus.cc:482] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:25.198349 30653 raft_consensus.cc:3037] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:01:25.198352 29625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 171351us
I20240501 20:01:25.070026 30645 raft_consensus.cc:504] T 010ceabf3e5a4b1c9330b80dade765bb P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:25.197242 29653 raft_consensus.cc:2454] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
I20240501 20:01:25.120829 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5525fc4b82a44b0991c3dd6098fde81" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.202282 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8edcd903224d48f889b46e1b5318e0ec" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:25.203970 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add9d7579351483fb268f89a44e95d8a" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:25.147080 29651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5345f6eb62449ba69b4a9c2e349b31" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.191644 30654 raft_consensus.cc:504] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:25.194849 29654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48fd0a8e289401aa7417c42ff2c81b2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.208492 29652 raft_consensus.cc:2454] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
I20240501 20:01:25.207664 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48630ae6eb394ac1b54955dd01cf0ed2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:25.209024 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a2bba62f434d1c8489b55c8fec2187" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:25.192454 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668b4a4d782c4278932dd8413dc3ab3e" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.209697 29801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36057b3f24d44b9aac3a1378053a395f" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:25.115348 30665 raft_consensus.cc:504] T 3c5345f6eb62449ba69b4a9c2e349b31 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:25.214535 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38cbeb0dc0c14e3d9182101eea885625" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.215145 29653 raft_consensus.cc:2454] T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
I20240501 20:01:25.213791 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38cbeb0dc0c14e3d9182101eea885625" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:25.198467 29802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d88139f89c47ff84cad674147aae0f" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:25.217919 30654 leader_election.cc:288] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:25.216132 29623 consensus_peers.cc:485] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f -> Peer 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227): Couldn't send request to peer 888d99522e4844118f8f1cf26f268c8c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: be383be783234855878cb8fd442c7b05. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:25.216639 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668b4a4d782c4278932dd8413dc3ab3e" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:25.216722 30385 leader_election.cc:302] T add9d7579351483fb268f89a44e95d8a P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 888d99522e4844118f8f1cf26f268c8c, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:25.219681 30658 raft_consensus.cc:2781] T add9d7579351483fb268f89a44e95d8a P 888d99522e4844118f8f1cf26f268c8c [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:01:25.216621 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfac893b4b4f457694a4eb184b6abb53" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:25.217571 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be383be783234855878cb8fd442c7b05" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:25.220193 30658 raft_consensus.cc:686] T add9d7579351483fb268f89a44e95d8a P 888d99522e4844118f8f1cf26f268c8c [term 2 LEADER]: Becoming Leader. State: Replica: 888d99522e4844118f8f1cf26f268c8c, State: Running, Role: LEADER
I20240501 20:01:25.220481 30418 raft_consensus.cc:1226] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [term 2 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:25.221213 30658 consensus_queue.cc:237] T add9d7579351483fb268f89a44e95d8a P 888d99522e4844118f8f1cf26f268c8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:25.225096 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48630ae6eb394ac1b54955dd01cf0ed2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:25.225493 30645 leader_election.cc:288] T 010ceabf3e5a4b1c9330b80dade765bb P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:25.226217 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8edcd903224d48f889b46e1b5318e0ec" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:25.227617 29802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfac893b4b4f457694a4eb184b6abb53" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:25.216013 29658 raft_consensus.cc:2454] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
I20240501 20:01:25.228894 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d86f8848a184a87947fd927c44e5a35" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.229064 29735 raft_consensus.cc:1226] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:25.229648 29657 raft_consensus.cc:2454] T 2d86f8848a184a87947fd927c44e5a35 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
I20240501 20:01:25.226783 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77be7101c0fe4085b6706296f609870a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:25.229785 30667 raft_consensus.cc:504] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:25.218616 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6797d82dda2948d8a7445fcd33e44752" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:25.231014 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be383be783234855878cb8fd442c7b05" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.229813 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b337b8e3e38d4ad6b1076fbd0d0a1609" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.231228 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e89c0572e4ee7be8e5315b459bdb3" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:25.231812 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e9f75387564275a60d38e9a910319c" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:25.212936 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77be7101c0fe4085b6706296f609870a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:25.227164 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a2bba62f434d1c8489b55c8fec2187" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:25.227262 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6797d82dda2948d8a7445fcd33e44752" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:25.218101 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31fff64b453a423eb0f567b3171564fb" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:25.232161 29732 raft_consensus.cc:2454] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:25.234465 29806 raft_consensus.cc:1226] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:25.235097 29653 raft_consensus.cc:2454] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
I20240501 20:01:25.236735 30646 consensus_queue.cc:1046] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240501 20:01:25.237080 29773 leader_election.cc:302] T 456035a1cb3f4f7da52d0fc1dd1c656c P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:25.236894 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e89c0572e4ee7be8e5315b459bdb3" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.238871 29773 leader_election.cc:302] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:25.239022 30654 raft_consensus.cc:3032] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:25.239439 30654 raft_consensus.cc:729] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:25.239719 30665 leader_election.cc:288] T 3c5345f6eb62449ba69b4a9c2e349b31 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:25.240264 30662 raft_consensus.cc:2781] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:25.240743 30662 raft_consensus.cc:482] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:25.240483 30654 consensus_queue.cc:260] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:25.241150 30662 raft_consensus.cc:3037] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:25.241708 30654 raft_consensus.cc:3037] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:25.239708 30613 raft_consensus.cc:2781] T 456035a1cb3f4f7da52d0fc1dd1c656c P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:25.244988 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfac893b4b4f457694a4eb184b6abb53" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:25.248283 29773 leader_election.cc:302] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:25.249202 29651 raft_consensus.cc:2379] T 3c5345f6eb62449ba69b4a9c2e349b31 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 2: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:25.249547 30614 raft_consensus.cc:2781] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:25.249923 30614 raft_consensus.cc:482] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:25.244437 30613 raft_consensus.cc:482] T 456035a1cb3f4f7da52d0fc1dd1c656c P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:25.250290 30614 raft_consensus.cc:3037] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:25.250444 30613 raft_consensus.cc:3037] T 456035a1cb3f4f7da52d0fc1dd1c656c P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:25.251523 30418 raft_consensus.cc:1226] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [term 2 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:25.252329 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec073fe645614fe28e7837de3c22510b" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.252526 29773 leader_election.cc:302] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:25.253355 30642 raft_consensus.cc:2781] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:25.253407 29773 leader_election.cc:302] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:25.253940 30642 raft_consensus.cc:482] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:25.254343 30642 raft_consensus.cc:3037] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:25.258625 30614 raft_consensus.cc:504] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } }
I20240501 20:01:25.260658 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38cbeb0dc0c14e3d9182101eea885625" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:25.261346 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31fff64b453a423eb0f567b3171564fb" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:25.261646 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b337b8e3e38d4ad6b1076fbd0d0a1609" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:25.262363 29802 raft_consensus.cc:1226] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:25.262629 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07bdb8b09b8c4ef4b8e49daa9b8c01d6" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:25.262949 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f261d2a05bf4c838532049601694b4a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:25.264194 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a35893b91247a99912eba787d787cf" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:25.264426 29802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece8d4ed84c34b748f7b6272b31c90ac" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:25.260404 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668b4a4d782c4278932dd8413dc3ab3e" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:25.253266 30646 consensus_queue.cc:1046] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240501 20:01:25.253516 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48fd0a8e289401aa7417c42ff2c81b2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:25.267254 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b337b8e3e38d4ad6b1076fbd0d0a1609" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:25.265964 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649b71f9f267437ca51c758dd9ca6987" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:25.268663 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458740ab5c444b6ab3d1991c69db5084" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:25.270146 29802 raft_consensus.cc:1261] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:01:25.270896 30655 raft_consensus.cc:3032] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:25.271059 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece8d4ed84c34b748f7b6272b31c90ac" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:25.271354 30655 raft_consensus.cc:729] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:25.271548 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07bdb8b09b8c4ef4b8e49daa9b8c01d6" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:25.272303 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0374bb790ed345ed90452595f71383a2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:25.272723 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458740ab5c444b6ab3d1991c69db5084" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:25.272341 30655 consensus_queue.cc:260] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:25.270795 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd5e83faaff4e2da4e4cc22ba410d33" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:25.273504 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a35893b91247a99912eba787d787cf" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:25.273810 30655 raft_consensus.cc:3037] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:25.274945 29735 raft_consensus.cc:1261] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:01:25.275136 30662 raft_consensus.cc:504] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:25.275578 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84fa28fb55f44b61ba1a88ed775a78ba" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:25.276286 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05df43bdfa479b902b1da297fb5974" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:25.277424 29656 raft_consensus.cc:1226] T 3c5345f6eb62449ba69b4a9c2e349b31 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:25.275998 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e07220db7444ae97f294d70eda6a93" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:25.278492 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f261d2a05bf4c838532049601694b4a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:25.278724 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb27193abbf4644a48662909bc7b04a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.279223 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd5e83faaff4e2da4e4cc22ba410d33" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.280359 29654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d2cac25f354a97b1bfb53d57ae7c08" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.280767 30614 leader_election.cc:288] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:25.281368 30614 raft_consensus.cc:2781] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:25.281765 30667 leader_election.cc:288] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 2 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:25.273834 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649b71f9f267437ca51c758dd9ca6987" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:25.282471 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ed27033ab1645e9ab0600ab635d9589" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:25.282721 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec073fe645614fe28e7837de3c22510b" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.282948 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e89c0572e4ee7be8e5315b459bdb3" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:25.283922 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5525f546426c47a38edf7f56ad04b7ab" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.281594 30418 raft_consensus.cc:1226] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [term 4 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 3. Current term is 4. Ops: []
I20240501 20:01:25.285156 30653 raft_consensus.cc:504] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:25.286557 30646 consensus_queue.cc:1046] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240501 20:01:25.288401 30662 leader_election.cc:288] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:25.281818 30614 raft_consensus.cc:482] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:25.291198 30664 raft_consensus.cc:3032] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:01:25.291402 30614 raft_consensus.cc:3037] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:25.291702 30664 raft_consensus.cc:729] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:25.292482 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1a08b455b94ae089396a15c837ddc2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.293030 30664 consensus_queue.cc:260] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:25.294072 29658 raft_consensus.cc:1226] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:25.294234 30664 raft_consensus.cc:3037] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:25.296739 29657 raft_consensus.cc:1226] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:25.297353 29655 raft_consensus.cc:1226] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:25.299566 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ed27033ab1645e9ab0600ab635d9589" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.306367 29659 raft_consensus.cc:1226] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:25.308022 29659 raft_consensus.cc:1226] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:25.316622 30614 raft_consensus.cc:504] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:25.307875 29651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca3948261374cef8b1b206a43c00173" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.320516 30653 leader_election.cc:288] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:25.322250 30384 leader_election.cc:302] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 888d99522e4844118f8f1cf26f268c8c; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:25.323673 30384 leader_election.cc:302] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 888d99522e4844118f8f1cf26f268c8c; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:25.324414 30667 raft_consensus.cc:2726] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 20:01:25.324854 30384 leader_election.cc:302] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 888d99522e4844118f8f1cf26f268c8c; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:25.325002 30667 raft_consensus.cc:2726] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 20:01:25.325664 30651 raft_consensus.cc:2726] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 20:01:25.326696 30384 leader_election.cc:302] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 888d99522e4844118f8f1cf26f268c8c; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:25.336228 30651 raft_consensus.cc:2726] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 20:01:25.331359 29655 raft_consensus.cc:2454] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 2.
I20240501 20:01:25.341737 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36057b3f24d44b9aac3a1378053a395f" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:25.342160 29802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e9f75387564275a60d38e9a910319c" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:25.343048 30642 raft_consensus.cc:504] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:25.345093 30642 leader_election.cc:288] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
W20240501 20:01:25.346369 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 188075us
I20240501 20:01:25.349380 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211e27a3b0084b89a6dd7550849b5a65" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.352658 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e07220db7444ae97f294d70eda6a93" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.353287 29652 raft_consensus.cc:1226] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:25.353425 29659 raft_consensus.cc:1226] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:25.354931 29624 leader_election.cc:302] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:25.355633 30664 raft_consensus.cc:2781] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:25.356119 30664 raft_consensus.cc:482] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:25.340637 30613 raft_consensus.cc:504] T 456035a1cb3f4f7da52d0fc1dd1c656c P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:25.356738 30664 raft_consensus.cc:3037] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:25.360178 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9472c53f79ec423f82c396bd980674e2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.360520 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be383be783234855878cb8fd442c7b05" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
W20240501 20:01:25.362627 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 109279us
I20240501 20:01:25.362849 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48630ae6eb394ac1b54955dd01cf0ed2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.360658 29557 catalog_manager.cc:5484] T add9d7579351483fb268f89a44e95d8a P 888d99522e4844118f8f1cf26f268c8c reported cstate change: term changed from 1 to 2, leader changed from ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) to 888d99522e4844118f8f1cf26f268c8c (127.27.217.196). New cstate: current_term: 2 leader_uuid: "888d99522e4844118f8f1cf26f268c8c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:01:25.365973 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5525fc4b82a44b0991c3dd6098fde81" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.366357 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c597f15e844f30837012baed4b0501" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
W20240501 20:01:25.366919 29772 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 96786.4us
I20240501 20:01:25.369526 29773 leader_election.cc:302] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:25.369793 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f5d1686214485d91a0baf4b870740e" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:25.371795 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:25.364702 30614 leader_election.cc:288] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:25.376358 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e49dd5180f04313ab33942fb3ddc147" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:25.373509 30613 leader_election.cc:288] T 456035a1cb3f4f7da52d0fc1dd1c656c P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:25.378491 30662 raft_consensus.cc:2781] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:25.379032 30662 raft_consensus.cc:482] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:25.379487 30662 raft_consensus.cc:3037] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:25.385825 29653 raft_consensus.cc:2454] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 2.
I20240501 20:01:25.389513 30662 raft_consensus.cc:504] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:25.396504 30662 leader_election.cc:288] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:25.397311 29624 leader_election.cc:302] T 010ceabf3e5a4b1c9330b80dade765bb P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
W20240501 20:01:25.410940 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.196:37227, user_credentials={real_user=slave}} blocked reactor thread for 119901us
I20240501 20:01:25.393790 30664 raft_consensus.cc:504] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:25.403043 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcf30cf66714efdac997862dafbe4a2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
W20240501 20:01:25.404865 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 46014.8us
I20240501 20:01:25.410902 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8cd40cb4044d18a86001ce6ddf97ab" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.414637 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77be7101c0fe4085b6706296f609870a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.414939 29624 leader_election.cc:302] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:25.415946 30655 raft_consensus.cc:2726] T 010ceabf3e5a4b1c9330b80dade765bb P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:25.417560 29698 leader_election.cc:302] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 888d99522e4844118f8f1cf26f268c8c, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:25.418262 30646 raft_consensus.cc:2726] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:25.419392 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c651044a305c4168805825a53d910899" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.417567 29624 leader_election.cc:302] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:25.420768 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69915dd5992641e3a24f7f45f1cd5c0c" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
W20240501 20:01:25.419533 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 53999.7us
I20240501 20:01:25.425091 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "456035a1cb3f4f7da52d0fc1dd1c656c" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.425175 29698 leader_election.cc:302] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 888d99522e4844118f8f1cf26f268c8c, fdfeac4190e045f290e003b2298d988d; no voters:
W20240501 20:01:25.428956 29772 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 60251.6us
I20240501 20:01:25.431028 29624 leader_election.cc:302] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:25.433076 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49da4246f5f6438abe0e7e8c9a523d80" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.428156 30655 raft_consensus.cc:2781] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:25.437367 30655 raft_consensus.cc:482] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:25.437786 30655 raft_consensus.cc:3037] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:25.435441 29802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010ceabf3e5a4b1c9330b80dade765bb" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:25.444968 29652 raft_consensus.cc:1226] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:25.446238 30646 raft_consensus.cc:2726] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:25.448145 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84fa28fb55f44b61ba1a88ed775a78ba" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.448516 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48fd0a8e289401aa7417c42ff2c81b2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.448588 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.450742 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d88139f89c47ff84cad674147aae0f" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.451643 30679 raft_consensus.cc:2781] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:25.452934 29654 raft_consensus.cc:1226] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:25.452675 30679 raft_consensus.cc:482] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:25.453773 30645 raft_consensus.cc:2781] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:25.453986 30679 raft_consensus.cc:3037] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:25.454260 30645 raft_consensus.cc:482] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:25.454749 30645 raft_consensus.cc:3037] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:25.457579 29624 leader_election.cc:302] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:25.463217 30664 leader_election.cc:288] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:25.464320 30654 raft_consensus.cc:2781] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:25.464707 30654 raft_consensus.cc:482] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:25.465068 30654 raft_consensus.cc:3037] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:25.468165 30662 consensus_queue.cc:1046] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:25.469862 30662 raft_consensus.cc:3032] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:25.470202 30662 raft_consensus.cc:729] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:25.470959 29653 raft_consensus.cc:1226] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:25.470993 30662 consensus_queue.cc:260] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:25.472533 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36057b3f24d44b9aac3a1378053a395f" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.472734 30662 raft_consensus.cc:3037] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:25.473484 29652 raft_consensus.cc:1226] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:25.474835 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e9f75387564275a60d38e9a910319c" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.446280 30655 raft_consensus.cc:504] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:25.477092 30655 leader_election.cc:288] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:25.474668 30679 raft_consensus.cc:504] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:25.479986 29700 leader_election.cc:302] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:25.479096 29773 leader_election.cc:302] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 888d99522e4844118f8f1cf26f268c8c, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:25.483445 29624 leader_election.cc:302] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:25.485267 30646 raft_consensus.cc:2726] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:25.483809 30681 raft_consensus.cc:2726] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:25.486220 30662 raft_consensus.cc:2726] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:25.486752 30679 leader_election.cc:288] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:25.487879 30645 raft_consensus.cc:504] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:25.488970 30654 raft_consensus.cc:504] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:25.491062 30654 leader_election.cc:288] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:25.493474 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.196:37227, user_credentials={real_user=slave}} blocked reactor thread for 68407us
I20240501 20:01:25.495815 29698 leader_election.cc:302] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 888d99522e4844118f8f1cf26f268c8c, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:25.495901 30645 leader_election.cc:288] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:25.492900 29772 leader_election.cc:302] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:25.501166 29772 leader_election.cc:302] T 84fa28fb55f44b61ba1a88ed775a78ba P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:25.501783 30662 raft_consensus.cc:3037] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:25.502254 29624 leader_election.cc:302] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:25.501875 30677 raft_consensus.cc:2726] T 84fa28fb55f44b61ba1a88ed775a78ba P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:25.503651 29772 leader_election.cc:302] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:25.503808 30674 raft_consensus.cc:2781] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:25.504248 30674 raft_consensus.cc:482] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:25.504326 30654 raft_consensus.cc:2726] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:25.504616 30674 raft_consensus.cc:3037] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:25.505044 30614 raft_consensus.cc:2726] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:25.510993 30662 raft_consensus.cc:2726] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:25.511088 30674 raft_consensus.cc:504] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:25.536036 30682 consensus_queue.cc:1046] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:25.537106 30679 raft_consensus.cc:3032] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:25.512557 29802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e89c0572e4ee7be8e5315b459bdb3" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:25.537600 30679 raft_consensus.cc:729] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:25.538003 29802 raft_consensus.cc:2379] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 888d99522e4844118f8f1cf26f268c8c in current term 2: Already voted for candidate ee0d97247d394a8facfa774f15f3a67d in this term.
I20240501 20:01:25.516187 30420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ed27033ab1645e9ab0600ab635d9589" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c"
I20240501 20:01:25.538434 30682 consensus_queue.cc:1046] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:25.516309 29700 leader_election.cc:302] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:25.516386 29772 leader_election.cc:302] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters: fdfeac4190e045f290e003b2298d988d
I20240501 20:01:25.539616 30681 raft_consensus.cc:3032] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:25.539757 30672 raft_consensus.cc:2781] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:25.539999 30681 raft_consensus.cc:729] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:25.540288 30672 raft_consensus.cc:482] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:25.540268 29772 leader_election.cc:302] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:25.540880 30672 raft_consensus.cc:3037] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:25.541359 30681 consensus_queue.cc:260] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:25.542730 29772 leader_election.cc:302] T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters: fdfeac4190e045f290e003b2298d988d
I20240501 20:01:25.519152 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69915dd5992641e3a24f7f45f1cd5c0c" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:25.520025 30676 consensus_queue.cc:1046] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:25.544858 30614 raft_consensus.cc:2781] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:25.544939 30676 raft_consensus.cc:3037] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:25.545295 30614 raft_consensus.cc:482] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:25.545768 30614 raft_consensus.cc:3037] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:25.520267 29624 leader_election.cc:302] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:25.548079 30673 raft_consensus.cc:2781] T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:25.548455 30673 raft_consensus.cc:482] T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:25.548787 30673 raft_consensus.cc:3037] T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:25.543768 29772 leader_election.cc:302] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters: fdfeac4190e045f290e003b2298d988d
I20240501 20:01:25.551702 30642 raft_consensus.cc:3032] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:25.552073 30642 raft_consensus.cc:729] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:25.521479 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c597f15e844f30837012baed4b0501" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:25.523787 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48fd0a8e289401aa7417c42ff2c81b2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:25.525583 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "456035a1cb3f4f7da52d0fc1dd1c656c" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:25.553079 30642 consensus_queue.cc:260] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:25.553613 30654 raft_consensus.cc:2781] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:25.554103 30654 raft_consensus.cc:482] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:25.553705 30672 raft_consensus.cc:504] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:25.554314 30642 raft_consensus.cc:3037] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:25.554514 30654 raft_consensus.cc:3037] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:25.526796 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:25.555850 30672 leader_election.cc:288] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:25.527444 29730 raft_consensus.cc:1226] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:25.556216 30646 consensus_queue.cc:1035] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:25.522555 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5525fc4b82a44b0991c3dd6098fde81" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:25.537166 30674 leader_election.cc:288] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Requested vote from peers ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:25.538519 30679 consensus_queue.cc:260] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:25.543522 30385 leader_election.cc:302] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 888d99522e4844118f8f1cf26f268c8c; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:25.559440 30679 raft_consensus.cc:3037] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:25.560065 30658 raft_consensus.cc:2726] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 20:01:25.529381 30645 consensus_queue.cc:1046] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 20:01:25.561167 30645 raft_consensus.cc:3032] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:25.561497 30645 raft_consensus.cc:729] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:25.562309 30645 consensus_queue.cc:260] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:25.563460 30645 raft_consensus.cc:3037] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:25.514622 30418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd5e83faaff4e2da4e4cc22ba410d33" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c"
I20240501 20:01:25.552021 29772 leader_election.cc:302] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters: fdfeac4190e045f290e003b2298d988d
I20240501 20:01:25.552569 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5345f6eb62449ba69b4a9c2e349b31" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:25.565438 29772 leader_election.cc:302] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters: fdfeac4190e045f290e003b2298d988d
I20240501 20:01:25.566150 30678 raft_consensus.cc:2781] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:25.565896 30614 raft_consensus.cc:504] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:25.566495 29772 leader_election.cc:302] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters: fdfeac4190e045f290e003b2298d988d
I20240501 20:01:25.566517 30678 raft_consensus.cc:482] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:25.567256 30678 raft_consensus.cc:3037] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:25.567305 30671 raft_consensus.cc:2781] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:25.567731 30671 raft_consensus.cc:482] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:25.567854 30614 leader_election.cc:288] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:25.568168 30671 raft_consensus.cc:3037] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:25.568507 30614 raft_consensus.cc:2781] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:25.568892 30614 raft_consensus.cc:482] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:25.569234 30614 raft_consensus.cc:3037] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:25.552695 30662 raft_consensus.cc:2781] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:25.571938 30662 raft_consensus.cc:482] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:25.572368 30662 raft_consensus.cc:3037] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:25.575266 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e38a2e91a3d48d09a22abce723af833" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:25.577122 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcf30cf66714efdac997862dafbe4a2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:25.578711 30614 raft_consensus.cc:504] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:25.579345 30662 raft_consensus.cc:504] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:25.580343 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5525f546426c47a38edf7f56ad04b7ab" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:25.581382 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49da4246f5f6438abe0e7e8c9a523d80" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:25.582343 30418 raft_consensus.cc:1226] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 2 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:25.584357 30676 raft_consensus.cc:2726] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:25.572094 29772 leader_election.cc:302] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters: fdfeac4190e045f290e003b2298d988d
I20240501 20:01:25.586783 30655 consensus_queue.cc:1046] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240501 20:01:25.596932 30655 raft_consensus.cc:3032] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:25.597306 30655 raft_consensus.cc:729] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:25.598642 30681 raft_consensus.cc:3037] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:25.598155 30655 consensus_queue.cc:260] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:25.589988 30654 raft_consensus.cc:504] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:25.588156 30671 raft_consensus.cc:504] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:25.590960 30662 leader_election.cc:288] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:25.586146 30614 leader_election.cc:288] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
W20240501 20:01:25.593602 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 55729.7us
I20240501 20:01:25.600939 30677 raft_consensus.cc:2781] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:25.601320 29772 leader_election.cc:302] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:25.607553 29698 leader_election.cc:302] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:25.607954 29772 leader_election.cc:302] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, 888d99522e4844118f8f1cf26f268c8c
I20240501 20:01:25.613054 29772 leader_election.cc:302] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters: fdfeac4190e045f290e003b2298d988d
I20240501 20:01:25.616142 29625 leader_election.cc:302] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
W20240501 20:01:25.617408 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 77557.1us
I20240501 20:01:25.617846 29625 leader_election.cc:302] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:25.618892 29625 leader_election.cc:302] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:25.614235 30662 raft_consensus.cc:2726] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:25.623279 29625 leader_election.cc:302] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:25.612639 30677 raft_consensus.cc:482] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:25.614737 30655 raft_consensus.cc:3037] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:25.615015 30679 raft_consensus.cc:2726] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:25.625218 29625 leader_election.cc:302] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:25.628752 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6797d82dda2948d8a7445fcd33e44752" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:25.617141 30613 raft_consensus.cc:2781] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:25.617506 30614 raft_consensus.cc:2726] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 20:01:25.613829 30673 raft_consensus.cc:504] T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:25.632380 30613 raft_consensus.cc:482] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:25.638151 30613 raft_consensus.cc:3037] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:25.638278 30673 leader_election.cc:288] T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:25.640396 30643 raft_consensus.cc:2726] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:25.625555 30677 raft_consensus.cc:3037] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:25.634428 29625 leader_election.cc:302] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:25.625154 30650 raft_consensus.cc:2726] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:25.620151 30624 raft_consensus.cc:2726] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:25.621018 30653 raft_consensus.cc:2726] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:25.647910 30643 raft_consensus.cc:2726] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:25.627861 30654 leader_election.cc:288] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:25.624701 30678 raft_consensus.cc:504] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:25.651459 29698 leader_election.cc:302] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:25.648749 29625 leader_election.cc:302] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:25.653512 29772 leader_election.cc:302] T ec073fe645614fe28e7837de3c22510b P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:25.651561 30685 consensus_queue.cc:1046] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 20:01:25.656294 30676 raft_consensus.cc:2726] T ec073fe645614fe28e7837de3c22510b P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:25.629639 30664 raft_consensus.cc:2726] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 20:01:25.658092 30654 raft_consensus.cc:2726] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 20:01:25.660272 29700 leader_election.cc:302] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:25.652115 30678 leader_election.cc:288] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:25.655217 29772 leader_election.cc:302] T 3c5345f6eb62449ba69b4a9c2e349b31 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:25.664345 30678 raft_consensus.cc:3037] T 3c5345f6eb62449ba69b4a9c2e349b31 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:25.670603 30685 raft_consensus.cc:3032] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:25.671053 30685 raft_consensus.cc:729] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:25.671350 29652 raft_consensus.cc:3037] T add9d7579351483fb268f89a44e95d8a P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:25.671914 30685 consensus_queue.cc:260] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:25.673552 30685 raft_consensus.cc:3037] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:25.680421 30679 raft_consensus.cc:2781] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:25.680860 30679 raft_consensus.cc:482] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:25.681281 30679 raft_consensus.cc:3037] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:25.682318 29698 leader_election.cc:302] T 1ca3948261374cef8b1b206a43c00173 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:25.682408 30674 consensus_queue.cc:1046] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 20:01:25.684469 29652 raft_consensus.cc:1261] T add9d7579351483fb268f89a44e95d8a P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Refusing update from remote peer 888d99522e4844118f8f1cf26f268c8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 20:01:25.690495 29625 leader_election.cc:302] T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:25.696884 30676 raft_consensus.cc:482] T a8787bc4d6b44649b9138d906730110a P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:25.674116 30671 leader_election.cc:288] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:25.674311 30681 raft_consensus.cc:2726] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:25.706210 29625 leader_election.cc:302] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:25.707604 29625 leader_election.cc:302] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:25.697546 30654 raft_consensus.cc:2726] T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:25.697476 30676 raft_consensus.cc:504] T a8787bc4d6b44649b9138d906730110a P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:25.711256 29625 leader_election.cc:302] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:25.712306 29625 leader_election.cc:302] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:25.713512 29625 leader_election.cc:302] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:25.707970 30664 raft_consensus.cc:2726] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 20:01:25.714118 30650 raft_consensus.cc:2726] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 20:01:25.712639 30643 raft_consensus.cc:2726] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 20:01:25.708698 30624 raft_consensus.cc:2726] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:25.706899 30673 raft_consensus.cc:482] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:25.712407 30654 consensus_queue.cc:1046] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20240501 20:01:25.715364 29625 leader_election.cc:302] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:25.715196 30673 raft_consensus.cc:504] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:25.715986 30654 raft_consensus.cc:3032] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:25.716361 30654 raft_consensus.cc:729] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:25.717211 30673 leader_election.cc:288] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:25.717654 30645 raft_consensus.cc:3032] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:25.717155 30654 consensus_queue.cc:260] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } }
I20240501 20:01:25.718014 30645 raft_consensus.cc:729] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:25.718647 30654 raft_consensus.cc:3037] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:25.718911 30645 consensus_queue.cc:260] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:25.720192 30646 raft_consensus.cc:2726] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 20:01:25.708844 30682 raft_consensus.cc:2726] T 1ca3948261374cef8b1b206a43c00173 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:25.720331 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010ceabf3e5a4b1c9330b80dade765bb" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.714368 29802 raft_consensus.cc:1261] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Refusing update from remote peer 888d99522e4844118f8f1cf26f268c8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 20:01:25.707983 30658 consensus_queue.cc:1035] T add9d7579351483fb268f89a44e95d8a P 888d99522e4844118f8f1cf26f268c8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.013s
I20240501 20:01:25.721616 30613 raft_consensus.cc:504] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:25.722458 30658 consensus_queue.cc:1035] T add9d7579351483fb268f89a44e95d8a P 888d99522e4844118f8f1cf26f268c8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:01:25.723606 30613 leader_election.cc:288] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:25.713493 30649 raft_consensus.cc:2726] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 20:01:25.727641 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69915dd5992641e3a24f7f45f1cd5c0c" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:25.728634 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48fd0a8e289401aa7417c42ff2c81b2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:25.729064 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:25.729736 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd5e83faaff4e2da4e4cc22ba410d33" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:25.730249 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5525fc4b82a44b0991c3dd6098fde81" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:25.730737 29657 raft_consensus.cc:1226] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:25.729827 29655 raft_consensus.cc:1226] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:25.731685 29625 leader_election.cc:302] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:25.733096 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "456035a1cb3f4f7da52d0fc1dd1c656c" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:25.733196 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ed27033ab1645e9ab0600ab635d9589" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:25.734724 29654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c597f15e844f30837012baed4b0501" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:25.735332 29656 raft_consensus.cc:1226] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:25.736025 29654 raft_consensus.cc:1226] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:25.736572 30665 raft_consensus.cc:2726] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 20:01:25.736927 30676 leader_election.cc:288] T a8787bc4d6b44649b9138d906730110a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:25.737042 30645 raft_consensus.cc:3037] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:25.736943 30687 raft_consensus.cc:482] T f6d88139f89c47ff84cad674147aae0f P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:25.740701 30687 raft_consensus.cc:504] T f6d88139f89c47ff84cad674147aae0f P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:25.742980 30687 leader_election.cc:288] T f6d88139f89c47ff84cad674147aae0f P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:25.717100 30678 raft_consensus.cc:2726] T 3c5345f6eb62449ba69b4a9c2e349b31 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:25.744002 29727 raft_consensus.cc:2454] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 2.
I20240501 20:01:25.745390 29658 raft_consensus.cc:2454] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 2.
I20240501 20:01:25.746609 29625 leader_election.cc:302] T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:25.747611 30649 raft_consensus.cc:2726] T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 20:01:25.748207 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 44604.9us
I20240501 20:01:25.749089 29625 leader_election.cc:302] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:25.750115 29700 leader_election.cc:302] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:25.752688 29659 raft_consensus.cc:2454] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 4.
I20240501 20:01:25.756359 30649 raft_consensus.cc:2726] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 20:01:25.756385 30674 raft_consensus.cc:2781] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:25.757395 30674 raft_consensus.cc:482] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:25.757784 30674 raft_consensus.cc:3037] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:01:25.760149 30686 log.cc:924] Time spent T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.155s user 0.001s sys 0.000s
I20240501 20:01:25.765311 30649 consensus_queue.cc:1035] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:25.765772 30679 raft_consensus.cc:504] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:25.768469 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668b4a4d782c4278932dd8413dc3ab3e" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:25.769441 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfac893b4b4f457694a4eb184b6abb53" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:25.769577 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac2171683884f6f97cdfeb0a656cb49" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:25.771587 30677 raft_consensus.cc:504] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:25.774488 29809 raft_consensus.cc:1226] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:25.774626 29800 raft_consensus.cc:1226] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:25.776728 30674 raft_consensus.cc:504] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:25.778640 30674 leader_election.cc:288] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:25.774503 30688 ts_tablet_manager.cc:902] T be383be783234855878cb8fd442c7b05 P 888d99522e4844118f8f1cf26f268c8c: Initiating tablet copy from peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959)
I20240501 20:01:25.780010 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3f29f0eeea4b119984634202e5d9cd" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:25.780500 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38cbeb0dc0c14e3d9182101eea885625" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:25.780422 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af08384057a4d85977481472f460168" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:25.782075 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1a08b455b94ae089396a15c837ddc2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:25.782364 29654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b337b8e3e38d4ad6b1076fbd0d0a1609" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:25.787890 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0374bb790ed345ed90452595f71383a2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:25.782892 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb27193abbf4644a48662909bc7b04a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:25.789757 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be383be783234855878cb8fd442c7b05" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:25.790081 29800 raft_consensus.cc:2454] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
I20240501 20:01:25.783041 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec073fe645614fe28e7837de3c22510b" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:25.788640 29801 raft_consensus.cc:1226] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:25.783629 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f5d1686214485d91a0baf4b870740e" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:25.787825 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "696121417ea9403990456cfc6fc4f416" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:25.795176 29807 raft_consensus.cc:1226] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:25.782732 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb27193abbf4644a48662909bc7b04a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.799268 29700 leader_election.cc:302] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:25.799409 29624 leader_election.cc:302] T 3c5345f6eb62449ba69b4a9c2e349b31 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:25.806816 29803 raft_consensus.cc:1226] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 3. Current term is 4. Ops: []
I20240501 20:01:25.807955 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5525f546426c47a38edf7f56ad04b7ab" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:25.767973 30679 leader_election.cc:288] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:25.826862 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5345f6eb62449ba69b4a9c2e349b31" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:25.826566 30646 raft_consensus.cc:2932] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
W20240501 20:01:25.834286 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 152066us
I20240501 20:01:25.832897 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "456035a1cb3f4f7da52d0fc1dd1c656c" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:25.828375 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49da4246f5f6438abe0e7e8c9a523d80" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:25.848686 29731 raft_consensus.cc:1226] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:25.848555 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8787bc4d6b44649b9138d906730110a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.848991 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d88139f89c47ff84cad674147aae0f" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:25.869300 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcf30cf66714efdac997862dafbe4a2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:25.849129 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6797d82dda2948d8a7445fcd33e44752" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
W20240501 20:01:25.869210 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 70011.8us
I20240501 20:01:25.876214 29698 leader_election.cc:302] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:25.877760 29700 leader_election.cc:302] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:25.879155 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e38a2e91a3d48d09a22abce723af833" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:25.879473 30682 raft_consensus.cc:2726] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:25.879880 29698 leader_election.cc:302] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:25.881361 29698 leader_election.cc:302] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:25.882268 30643 consensus_queue.cc:1046] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:25.885049 29653 raft_consensus.cc:2454] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 2.
I20240501 20:01:25.846002 30665 raft_consensus.cc:2781] T 3c5345f6eb62449ba69b4a9c2e349b31 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:01:25.889022 30690 raft_consensus.cc:2726] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:25.889163 30673 consensus_queue.cc:1046] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 20:01:25.883311 30679 consensus_queue.cc:1046] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
I20240501 20:01:25.883210 30624 consensus_queue.cc:1046] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.007s
I20240501 20:01:25.849028 30645 raft_consensus.cc:2726] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:25.912309 30677 leader_election.cc:288] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:25.897914 30682 consensus_queue.cc:1046] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.015s
I20240501 20:01:25.912851 30679 raft_consensus.cc:3032] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:25.913251 30679 raft_consensus.cc:729] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:25.914011 30679 consensus_queue.cc:260] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:25.915266 30679 raft_consensus.cc:3037] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:25.916416 30690 raft_consensus.cc:3032] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:25.916785 30690 raft_consensus.cc:729] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:25.917593 30690 consensus_queue.cc:260] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:25.919302 30678 raft_consensus.cc:3032] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:25.919701 30678 raft_consensus.cc:729] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:25.901523 30676 consensus_queue.cc:1046] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20240501 20:01:25.920598 30671 consensus_queue.cc:1046] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 20:01:25.920553 30678 consensus_queue.cc:260] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:25.921676 30671 raft_consensus.cc:3032] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:25.921831 30678 raft_consensus.cc:3037] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:25.922063 30671 raft_consensus.cc:729] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:25.922370 30676 raft_consensus.cc:3032] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:25.922777 30676 raft_consensus.cc:729] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:25.922940 30671 consensus_queue.cc:260] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:25.896809 29731 raft_consensus.cc:1226] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:25.898860 29698 leader_election.cc:302] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:25.895143 30674 raft_consensus.cc:2726] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:25.902770 30672 raft_consensus.cc:2781] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:25.900033 30689 consensus_queue.cc:1046] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
I20240501 20:01:25.929515 29698 leader_election.cc:302] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
W20240501 20:01:25.931106 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 56064.5us
I20240501 20:01:25.936955 30695 consensus_queue.cc:1046] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:25.905354 30665 raft_consensus.cc:686] T 3c5345f6eb62449ba69b4a9c2e349b31 P 4c8da1d580614449a636b855bf11b95f [term 2 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:25.946182 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d86f8848a184a87947fd927c44e5a35" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:25.924935 29727 raft_consensus.cc:1226] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:25.947890 29772 leader_election.cc:302] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:25.923522 30676 consensus_queue.cc:260] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:25.928575 30672 raft_consensus.cc:482] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:25.905556 30613 consensus_queue.cc:1046] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.044s
I20240501 20:01:25.933404 30645 raft_consensus.cc:2726] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:25.967502 29698 leader_election.cc:302] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:25.953677 30614 raft_consensus.cc:2781] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:01:25.972617 29625 leader_election.cc:302] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:25.979521 30693 raft_consensus.cc:2726] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:25.984510 30689 consensus_queue.cc:1046] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 20:01:25.986407 30665 consensus_queue.cc:237] T 3c5345f6eb62449ba69b4a9c2e349b31 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:25.994768 29625 leader_election.cc:302] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:25.996119 29625 leader_election.cc:302] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:25.965584 30672 raft_consensus.cc:3037] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:01:26.015955 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 48582us
I20240501 20:01:26.017719 29625 leader_election.cc:302] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:25.994227 30650 raft_consensus.cc:3032] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:26.021657 29698 leader_election.cc:302] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:25.997596 30614 raft_consensus.cc:686] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [term 2 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:26.037950 29698 leader_election.cc:302] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:26.021195 30649 raft_consensus.cc:2726] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 20:01:26.026067 30676 raft_consensus.cc:3037] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:26.021988 30650 raft_consensus.cc:729] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:26.043036 30614 consensus_queue.cc:237] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } }
I20240501 20:01:26.023504 30703 raft_consensus.cc:2726] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 20:01:25.973482 30701 consensus_queue.cc:1046] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.060s
I20240501 20:01:26.050681 29698 leader_election.cc:302] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: 888d99522e4844118f8f1cf26f268c8c
I20240501 20:01:26.043125 30645 raft_consensus.cc:2726] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 20:01:26.041464 30698 consensus_queue.cc:1046] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.018s
I20240501 20:01:26.023907 30685 consensus_queue.cc:1046] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20240501 20:01:26.049388 30649 raft_consensus.cc:2726] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 20:01:26.066052 29698 leader_election.cc:302] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:26.071321 30650 consensus_queue.cc:260] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:26.053066 30708 raft_consensus.cc:2726] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 20:01:26.082356 30682 raft_consensus.cc:2726] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:26.058997 30689 raft_consensus.cc:2726] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 20:01:26.084751 29698 leader_election.cc:302] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:26.093955 29698 leader_election.cc:302] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:26.095454 29698 leader_election.cc:302] T c651044a305c4168805825a53d910899 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:26.065465 30703 raft_consensus.cc:2781] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:26.066715 30695 raft_consensus.cc:2726] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 20:01:26.124964 30689 raft_consensus.cc:2726] T c651044a305c4168805825a53d910899 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:26.125277 29807 raft_consensus.cc:1226] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:26.124294 29698 leader_election.cc:302] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:26.130916 29698 leader_election.cc:302] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:26.098274 30681 raft_consensus.cc:2726] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:26.126152 30682 raft_consensus.cc:2726] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:26.164798 30681 raft_consensus.cc:2726] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:26.126601 30703 raft_consensus.cc:482] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:26.180928 30671 raft_consensus.cc:3037] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:26.181757 30655 raft_consensus.cc:3032] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:26.198850 30690 raft_consensus.cc:3037] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:26.199976 30655 raft_consensus.cc:729] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:26.201149 30655 consensus_queue.cc:260] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:26.182569 30645 raft_consensus.cc:3032] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:26.184624 30649 raft_consensus.cc:3032] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:01:26.185353 30708 raft_consensus.cc:3032] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:26.167385 30674 raft_consensus.cc:3032] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:26.142716 30689 raft_consensus.cc:2726] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:26.189262 30681 consensus_queue.cc:1046] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:26.214788 30674 raft_consensus.cc:729] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:26.192720 30693 raft_consensus.cc:3032] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:26.215350 30679 raft_consensus.cc:3032] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:26.211410 30708 raft_consensus.cc:729] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:26.215870 30679 raft_consensus.cc:729] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:26.215647 30693 raft_consensus.cc:729] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:26.216850 30708 consensus_queue.cc:260] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:26.216987 30679 consensus_queue.cc:260] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:26.217172 30693 consensus_queue.cc:260] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:26.218200 30708 raft_consensus.cc:3037] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:26.219117 30693 raft_consensus.cc:3037] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:26.204327 30645 raft_consensus.cc:729] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:26.225672 30645 consensus_queue.cc:260] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:26.211885 30649 raft_consensus.cc:729] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:26.227977 30649 consensus_queue.cc:260] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:26.229728 30649 raft_consensus.cc:3037] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:26.215710 30674 consensus_queue.cc:260] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:26.233477 30674 raft_consensus.cc:3037] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:26.242590 30655 raft_consensus.cc:3037] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:26.245476 30676 raft_consensus.cc:482] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:26.246208 30676 raft_consensus.cc:504] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:26.250978 30676 leader_election.cc:288] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:26.256379 30676 raft_consensus.cc:482] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:26.257220 30678 raft_consensus.cc:482] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:26.257741 30678 raft_consensus.cc:504] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:26.261113 30678 leader_election.cc:288] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:26.261157 30687 raft_consensus.cc:482] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:26.256938 30676 raft_consensus.cc:504] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:26.262055 30687 raft_consensus.cc:504] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:26.264482 30687 leader_election.cc:288] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:26.265522 30676 leader_election.cc:288] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:26.202850 30703 raft_consensus.cc:3037] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:26.329023 30671 raft_consensus.cc:482] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
W20240501 20:01:26.358814 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 382271us
I20240501 20:01:26.365628 29809 raft_consensus.cc:1226] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:26.367450 29807 raft_consensus.cc:1226] T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:26.362607 30678 raft_consensus.cc:482] T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:26.368626 30678 raft_consensus.cc:504] T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:26.370612 30678 leader_election.cc:288] T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:26.357117 30671 raft_consensus.cc:504] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:26.383183 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 431892us
W20240501 20:01:26.383538 29625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 366020us
I20240501 20:01:26.384502 29625 leader_election.cc:302] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:26.399480 30693 consensus_queue.cc:1046] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:26.399302 30703 raft_consensus.cc:504] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:26.400962 30693 raft_consensus.cc:3032] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:26.407428 30646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:37932 (request call id 4) took 1589 ms. Trace:
I20240501 20:01:26.407953 29625 leader_election.cc:302] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:26.408736 30654 raft_consensus.cc:2726] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 20:01:26.410159 30672 raft_consensus.cc:504] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:26.411339 30665 consensus_queue.cc:1046] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:26.414588 29809 raft_consensus.cc:1226] T 456035a1cb3f4f7da52d0fc1dd1c656c P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:26.412339 30711 raft_consensus.cc:482] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:26.415813 30711 raft_consensus.cc:504] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:26.395637 30664 raft_consensus.cc:2726] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20240501 20:01:26.419008 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 262413us
I20240501 20:01:26.401212 30671 leader_election.cc:288] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:26.419950 29698 leader_election.cc:302] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:26.421069 29731 raft_consensus.cc:1226] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:26.423974 29802 raft_consensus.cc:2932] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
I20240501 20:01:26.404392 30693 raft_consensus.cc:729] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:26.427413 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb27193abbf4644a48662909bc7b04a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:26.428788 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e38a2e91a3d48d09a22abce723af833" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:26.429528 29659 raft_consensus.cc:2379] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 2: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:26.429448 29698 leader_election.cc:302] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:26.430598 29698 leader_election.cc:302] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:26.430882 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07bdb8b09b8c4ef4b8e49daa9b8c01d6" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:26.431270 30689 raft_consensus.cc:2726] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:26.431655 29659 raft_consensus.cc:2379] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 2: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:26.433303 29801 raft_consensus.cc:1226] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:26.418480 30649 raft_consensus.cc:3032] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:26.435878 30682 raft_consensus.cc:2726] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:26.436213 30649 raft_consensus.cc:729] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:26.437052 30649 consensus_queue.cc:260] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:26.408000 30646 rpcz_store.cc:276] 0501 20:01:24.817719 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:24.817875 (+ 156us) service_pool.cc:225] Handling call
0501 20:01:24.838343 (+ 20468us) spinlock_profiling.cc:243] Waited 12.4 ms on lock 0x7b2c00042f48. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe45625df11 00007fe45626cbda 00007fe45626c863 00007fe45626c801 00007fe4578b6ec2 00007fe4578b6c9d 00007fe4578821d4 00007fe457881993 00007fe45c7bece9 00007fe455ea6714 00007fe455ea66a2
0501 20:01:24.913341 (+ 74998us) log.cc:842] Serialized 513 byte log entry
0501 20:01:25.195141 (+281800us) spinlock_profiling.cc:243] Waited 1.6 ms on lock 0x7b1800136480. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe45d7f2661 00007fe45d7ff3e1 00007fe4548ca2ea 00007fe45dfb9fe5 00007fe45494e3d4 00007fe45494dcc5 00007fe4577f5570 00007fe45785b363 00007fe457881a32 00007fe45c7bece9 00007fe455ea6714
0501 20:01:26.407349 (+1212208us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":39378,"raft.queue_time_us":198682,"raft.run_cpu_time_us":32329,"raft.run_wall_time_us":180998,"spinlock_wait_cycles":33851264,"thread_start_us":29295,"threads_started":1,"wal-append.queue_time_us":118020,"wal-append.run_cpu_time_us":2149,"wal-append.run_wall_time_us":1029820}
I20240501 20:01:26.438400 30649 raft_consensus.cc:3037] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:26.439663 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458740ab5c444b6ab3d1991c69db5084" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:26.439987 30706 raft_consensus.cc:3032] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:26.440757 29653 raft_consensus.cc:2379] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 2: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:26.433817 29698 leader_election.cc:302] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:26.431679 29807 raft_consensus.cc:1226] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:26.427322 30672 leader_election.cc:288] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:26.434715 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece8d4ed84c34b748f7b6272b31c90ac" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:26.434981 30690 raft_consensus.cc:2726] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:26.442898 29652 raft_consensus.cc:2379] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 2: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:26.439657 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f261d2a05bf4c838532049601694b4a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:26.443763 29659 raft_consensus.cc:2379] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 2: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:26.440521 30706 raft_consensus.cc:729] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:26.445333 30682 consensus_queue.cc:1046] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 20:01:26.448297 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0374bb790ed345ed90452595f71383a2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:26.449368 29659 raft_consensus.cc:2379] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 2: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:26.452374 30690 raft_consensus.cc:3032] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:26.452574 30645 raft_consensus.cc:3037] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:26.452999 30690 raft_consensus.cc:729] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:26.454172 30690 consensus_queue.cc:260] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:26.455727 30690 raft_consensus.cc:3037] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:26.456147 30672 raft_consensus.cc:2726] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:26.456815 30650 raft_consensus.cc:3037] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:26.426990 30654 consensus_queue.cc:1046] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20240501 20:01:26.456277 30693 consensus_queue.cc:260] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:26.472764 30672 consensus_queue.cc:1046] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:26.473295 30703 leader_election.cc:288] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:26.473870 30703 raft_consensus.cc:3032] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:26.474210 30703 raft_consensus.cc:729] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:26.474350 30662 raft_consensus.cc:3032] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:26.474934 30662 raft_consensus.cc:729] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:26.474983 30703 consensus_queue.cc:260] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:26.476164 30662 consensus_queue.cc:260] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:26.457834 30711 leader_election.cc:288] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:26.445508 30706 consensus_queue.cc:260] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:26.480899 30706 raft_consensus.cc:3037] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:26.482934 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8edcd903224d48f889b46e1b5318e0ec" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:26.483570 29652 raft_consensus.cc:2379] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 2: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:26.484869 30654 raft_consensus.cc:3032] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:26.485178 30654 raft_consensus.cc:729] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:26.485738 30654 consensus_queue.cc:260] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:26.486526 30654 raft_consensus.cc:3037] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:26.486774 30679 raft_consensus.cc:3037] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:26.491196 29698 leader_election.cc:302] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:26.483131 30693 raft_consensus.cc:3037] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:26.496608 30688 tablet_copy_client.cc:308] T be383be783234855878cb8fd442c7b05 P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Beginning tablet copy session from remote peer at address 127.27.217.193:33959
I20240501 20:01:26.490955 30703 raft_consensus.cc:3037] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:26.498564 29670 tablet_copy_service.cc:140] P 4c8da1d580614449a636b855bf11b95f: Received BeginTabletCopySession request for tablet be383be783234855878cb8fd442c7b05 from peer 888d99522e4844118f8f1cf26f268c8c ({username='slave'} at 127.0.0.1:37936)
I20240501 20:01:26.482192 30681 consensus_queue.cc:1046] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20240501 20:01:26.504343 29670 tablet_copy_service.cc:161] P 4c8da1d580614449a636b855bf11b95f: Beginning new tablet copy session on tablet be383be783234855878cb8fd442c7b05 from peer 888d99522e4844118f8f1cf26f268c8c at {username='slave'} at 127.0.0.1:37936: session id = 888d99522e4844118f8f1cf26f268c8c-be383be783234855878cb8fd442c7b05
I20240501 20:01:26.505918 30682 consensus_queue.cc:1046] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 20:01:26.509673 30681 raft_consensus.cc:3032] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:26.510146 30681 raft_consensus.cc:729] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:26.510984 30681 consensus_queue.cc:260] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:26.493610 30672 raft_consensus.cc:2726] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:26.513118 30685 raft_consensus.cc:3032] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:26.513571 30685 raft_consensus.cc:729] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:26.515540 30685 consensus_queue.cc:260] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:26.516973 30685 raft_consensus.cc:3037] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:26.524152 30662 raft_consensus.cc:3037] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:26.519454 29557 catalog_manager.cc:5484] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 1 to 2, leader changed from 4c8da1d580614449a636b855bf11b95f (127.27.217.193) to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 2 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } }
W20240501 20:01:26.492661 30713 consensus_queue.cc:1159] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "fdfeac4190e045f290e003b2298d988d" responder_term: 2 status { last_received { term: 1 index: 1 } last_committed_idx: 1 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 1. Current term is 2. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20240501 20:01:26.541581 29670 tablet_copy_source_session.cc:205] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:01:26.543689 30681 raft_consensus.cc:3037] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:26.548511 29556 catalog_manager.cc:5484] T 3c5345f6eb62449ba69b4a9c2e349b31 P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 1 to 2, leader changed from fdfeac4190e045f290e003b2298d988d (127.27.217.194) to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 2 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:26.550391 29556 catalog_manager.cc:5484] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f reported cstate change: config changed from index -1 to 2, term changed from 1 to 2, leader changed from 4c8da1d580614449a636b855bf11b95f (127.27.217.193) to <none>, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New cstate: current_term: 2 committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
I20240501 20:01:26.554415 29698 leader_election.cc:302] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:26.554878 30706 consensus_queue.cc:1046] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 20:01:26.555672 30688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be383be783234855878cb8fd442c7b05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:26.555976 29698 leader_election.cc:302] T 010ceabf3e5a4b1c9330b80dade765bb P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:26.556859 30645 raft_consensus.cc:2726] T 010ceabf3e5a4b1c9330b80dade765bb P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:26.555272 30703 raft_consensus.cc:2726] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:26.559237 30713 raft_consensus.cc:3032] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:26.559597 30713 raft_consensus.cc:729] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:26.557891 29698 leader_election.cc:302] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:26.560369 30713 consensus_queue.cc:260] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:26.563519 29698 leader_election.cc:302] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:26.565197 29731 raft_consensus.cc:1226] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:26.563028 30685 raft_consensus.cc:2781] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:01:26.571501 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be383be783234855878cb8fd442c7b05" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:26.572906 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb27193abbf4644a48662909bc7b04a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:26.573418 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "456035a1cb3f4f7da52d0fc1dd1c656c" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:26.570595 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec073fe645614fe28e7837de3c22510b" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:26.575563 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e07220db7444ae97f294d70eda6a93" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:26.578140 29773 leader_election.cc:302] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:26.579010 30711 raft_consensus.cc:2726] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 20:01:26.579355 29773 leader_election.cc:302] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:26.567807 30713 raft_consensus.cc:3037] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:26.582101 29773 leader_election.cc:302] T 456035a1cb3f4f7da52d0fc1dd1c656c P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:26.581166 30642 raft_consensus.cc:2726] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 20:01:26.585561 29659 raft_consensus.cc:2454] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 2.
I20240501 20:01:26.586417 30420 raft_consensus.cc:1226] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [term 2 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:26.587755 29773 leader_election.cc:302] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:26.581226 30703 raft_consensus.cc:2781] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:01:26.594924 29653 raft_consensus.cc:2454] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 2.
I20240501 20:01:26.584429 30671 raft_consensus.cc:2726] T 456035a1cb3f4f7da52d0fc1dd1c656c P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 20:01:26.591647 30419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e07220db7444ae97f294d70eda6a93" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c"
I20240501 20:01:26.595674 30703 raft_consensus.cc:686] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:26.600726 30419 raft_consensus.cc:2379] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d in current term 2: Already voted for candidate 888d99522e4844118f8f1cf26f268c8c in this term.
I20240501 20:01:26.601007 30703 consensus_queue.cc:237] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:26.597781 30642 raft_consensus.cc:2726] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 20:01:26.599396 29802 raft_consensus.cc:1226] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:26.599598 29805 raft_consensus.cc:1226] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:26.585289 30418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "696121417ea9403990456cfc6fc4f416" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c"
I20240501 20:01:26.606300 30679 consensus_queue.cc:1046] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:26.607306 29657 raft_consensus.cc:2454] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 2.
I20240501 20:01:26.607786 29773 leader_election.cc:302] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:26.610227 29656 raft_consensus.cc:2454] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 2.
I20240501 20:01:26.611085 30679 consensus_queue.cc:1046] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:26.584767 30685 raft_consensus.cc:686] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d [term 2 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:26.611732 30642 consensus_queue.cc:1046] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:26.611726 30693 raft_consensus.cc:3032] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:26.624132 30673 raft_consensus.cc:3032] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:26.620496 29805 raft_consensus.cc:1226] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:26.625388 30685 consensus_queue.cc:237] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:26.621510 29802 raft_consensus.cc:1226] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:26.627293 30673 raft_consensus.cc:729] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:26.628300 30673 consensus_queue.cc:260] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:26.629768 30673 raft_consensus.cc:3037] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:26.631554 30645 consensus_queue.cc:1046] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:01:26.633013 30679 raft_consensus.cc:3032] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:26.633338 30679 raft_consensus.cc:729] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:26.633944 30679 consensus_queue.cc:260] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:26.634728 30679 raft_consensus.cc:3037] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:26.636322 30672 consensus_queue.cc:1046] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:26.636665 30690 raft_consensus.cc:3032] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:26.636749 30682 consensus_queue.cc:1046] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 20:01:26.637449 30672 raft_consensus.cc:3032] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:26.637835 30672 raft_consensus.cc:729] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:26.637943 30682 raft_consensus.cc:3032] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:26.638371 30682 raft_consensus.cc:729] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:26.638635 30672 consensus_queue.cc:260] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:26.639168 30682 consensus_queue.cc:260] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:26.640084 30672 raft_consensus.cc:3037] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:26.640772 30682 raft_consensus.cc:3037] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:26.624249 30693 raft_consensus.cc:729] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:26.643211 30693 consensus_queue.cc:260] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:26.644335 30693 raft_consensus.cc:3037] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:26.647398 29556 catalog_manager.cc:5484] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 2 to 1, leader changed from <none> to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 1 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
I20240501 20:01:26.637086 30690 raft_consensus.cc:729] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:26.650511 30690 consensus_queue.cc:260] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:26.658308 29656 raft_consensus.cc:1261] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:01:26.621927 30662 raft_consensus.cc:2726] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 20:01:26.660107 30690 raft_consensus.cc:3037] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:26.661041 30662 consensus_queue.cc:1035] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:26.672793 29660 raft_consensus.cc:2454] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 2.
W20240501 20:01:26.677320 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.196:37227, user_credentials={real_user=slave}} blocked reactor thread for 54146.1us
I20240501 20:01:26.681454 29698 leader_election.cc:302] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 888d99522e4844118f8f1cf26f268c8c, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:26.616884 30688 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:40658 (request call id 56) took 1009 ms. Trace:
I20240501 20:01:26.683571 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac2171683884f6f97cdfeb0a656cb49" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:26.684283 30672 raft_consensus.cc:2726] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 20:01:26.686131 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668b4a4d782c4278932dd8413dc3ab3e" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:26.691219 29727 raft_consensus.cc:2454] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 2.
I20240501 20:01:26.693770 29730 raft_consensus.cc:2454] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 2.
I20240501 20:01:26.696614 29554 catalog_manager.cc:5484] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 1 to 2, leader changed from 4c8da1d580614449a636b855bf11b95f (127.27.217.193) to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 2 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:26.683892 30688 rpcz_store.cc:276] 0501 20:01:25.620268 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:25.621886 (+ 1618us) service_pool.cc:225] Handling call
0501 20:01:25.796297 (+174411us) ts_tablet_manager.cc:903] T be383be783234855878cb8fd442c7b05 P 888d99522e4844118f8f1cf26f268c8c: Initiating tablet copy from peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959)
0501 20:01:26.545180 (+748883us) tablet_copy_client.cc:353] Tablet copy session begun
0501 20:01:26.598472 (+ 53292us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:26.598842 (+ 370us) tablet_copy_client.cc:482] Wrote new tablet metadata
0501 20:01:26.615786 (+ 16944us) tablet_copy_client.cc:403] Wrote new consensus metadata
0501 20:01:26.616222 (+ 436us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:26.616697 (+ 475us) ts_tablet_manager.cc:958] Replica registered.
0501 20:01:26.616782 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":5042,"tablet-copy.queue_time_us":151899,"thread_start_us":744689,"threads_started":5}
I20240501 20:01:26.699402 30688 tablet_copy_client.cc:783] T be383be783234855878cb8fd442c7b05 P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Starting download of 0 data blocks...
I20240501 20:01:26.700314 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfac893b4b4f457694a4eb184b6abb53" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:26.704617 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3f29f0eeea4b119984634202e5d9cd" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:26.707124 29733 raft_consensus.cc:2454] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 2.
I20240501 20:01:26.710506 29730 raft_consensus.cc:2454] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 2.
I20240501 20:01:26.712442 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af08384057a4d85977481472f460168" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:26.713114 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb27193abbf4644a48662909bc7b04a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:26.713981 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8787bc4d6b44649b9138d906730110a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:26.714447 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d88139f89c47ff84cad674147aae0f" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:26.716143 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b337b8e3e38d4ad6b1076fbd0d0a1609" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:26.718953 29727 raft_consensus.cc:1226] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:26.719924 29731 raft_consensus.cc:2454] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 2.
I20240501 20:01:26.721124 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38cbeb0dc0c14e3d9182101eea885625" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:26.722324 29733 raft_consensus.cc:2454] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 2.
I20240501 20:01:26.723067 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d86f8848a184a87947fd927c44e5a35" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:26.727425 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f261d2a05bf4c838532049601694b4a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:26.728083 29727 raft_consensus.cc:2454] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 2.
I20240501 20:01:26.728971 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0374bb790ed345ed90452595f71383a2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:26.730687 29726 raft_consensus.cc:3037] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:26.730808 29730 raft_consensus.cc:2454] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 2.
I20240501 20:01:26.725014 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e38a2e91a3d48d09a22abce723af833" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:26.725381 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07bdb8b09b8c4ef4b8e49daa9b8c01d6" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:26.732254 29731 raft_consensus.cc:2454] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 2.
I20240501 20:01:26.726066 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece8d4ed84c34b748f7b6272b31c90ac" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:26.733206 29735 raft_consensus.cc:2454] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 2.
I20240501 20:01:26.726405 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8edcd903224d48f889b46e1b5318e0ec" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:26.734046 29734 raft_consensus.cc:2454] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 2.
I20240501 20:01:26.726728 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458740ab5c444b6ab3d1991c69db5084" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:26.732589 29728 raft_consensus.cc:2454] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 2.
I20240501 20:01:26.734792 29729 raft_consensus.cc:2454] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 2.
I20240501 20:01:26.734879 29732 raft_consensus.cc:2454] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 2.
I20240501 20:01:26.735538 29773 leader_election.cc:302] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:26.741062 29727 raft_consensus.cc:2454] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 2.
I20240501 20:01:26.713156 30688 tablet_copy_client.cc:647] T be383be783234855878cb8fd442c7b05 P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Starting download of 1 WAL segments...
I20240501 20:01:26.741652 30673 raft_consensus.cc:2781] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:01:26.743022 30673 raft_consensus.cc:686] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [term 2 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:26.740478 29698 leader_election.cc:302] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:26.743469 29554 catalog_manager.cc:5484] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 3 to 4, leader changed from 4c8da1d580614449a636b855bf11b95f (127.27.217.193) to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 4 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:26.744510 30685 raft_consensus.cc:2781] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:01:26.744856 29698 leader_election.cc:302] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:26.745033 30685 raft_consensus.cc:686] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [term 2 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:26.745846 30685 consensus_queue.cc:237] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:26.748425 29726 raft_consensus.cc:1261] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:01:26.748311 30703 raft_consensus.cc:2781] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:01:26.749446 30703 raft_consensus.cc:686] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [term 2 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:26.749804 30688 tablet_copy_client.cc:523] T be383be783234855878cb8fd442c7b05 P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:01:26.750311 30703 consensus_queue.cc:237] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:26.745915 29698 leader_election.cc:302] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: 888d99522e4844118f8f1cf26f268c8c
I20240501 20:01:26.754740 29698 leader_election.cc:302] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:26.754936 30642 raft_consensus.cc:482] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:26.755404 30642 raft_consensus.cc:504] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:26.755822 29698 leader_election.cc:302] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:26.756542 30645 raft_consensus.cc:2781] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:01:26.757002 30645 raft_consensus.cc:686] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [term 2 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:26.757334 30642 leader_election.cc:288] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:26.757825 30645 consensus_queue.cc:237] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:26.758606 30682 raft_consensus.cc:2781] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:01:26.759372 30682 raft_consensus.cc:686] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d [term 2 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:26.759941 30693 raft_consensus.cc:2781] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:01:26.760465 30693 raft_consensus.cc:686] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d [term 2 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:26.760298 30682 consensus_queue.cc:237] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:26.762079 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e49dd5180f04313ab33942fb3ddc147" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:26.761312 30693 consensus_queue.cc:237] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:26.763379 29802 raft_consensus.cc:1261] T 3c5345f6eb62449ba69b4a9c2e349b31 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:01:26.763487 29773 leader_election.cc:302] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:26.764345 30662 raft_consensus.cc:2781] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:01:26.764710 29773 leader_election.cc:302] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:26.765619 30671 raft_consensus.cc:2781] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:01:26.766110 30671 raft_consensus.cc:686] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d [term 2 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:26.764792 30662 raft_consensus.cc:686] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d [term 2 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:26.767427 29773 leader_election.cc:302] T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:26.768777 29726 raft_consensus.cc:1261] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:01:26.766433 30706 consensus_queue.cc:1035] T 3c5345f6eb62449ba69b4a9c2e349b31 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20240501 20:01:26.770211 29773 leader_election.cc:302] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:26.771674 29773 leader_election.cc:302] T a8787bc4d6b44649b9138d906730110a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:26.776373 30650 consensus_queue.cc:1035] T 3c5345f6eb62449ba69b4a9c2e349b31 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 20:01:26.779340 29773 leader_election.cc:302] T f6d88139f89c47ff84cad674147aae0f P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:26.780767 30613 raft_consensus.cc:2726] T f6d88139f89c47ff84cad674147aae0f P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 20:01:26.781026 29606 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica fdfeac4190e045f290e003b2298d988d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } }
I20240501 20:01:26.781862 30662 consensus_queue.cc:237] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:26.775178 30713 raft_consensus.cc:482] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:26.769294 30671 consensus_queue.cc:237] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:26.787609 30713 raft_consensus.cc:504] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:26.775307 30614 raft_consensus.cc:2781] T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:01:26.778820 30642 raft_consensus.cc:2726] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:26.780018 30711 raft_consensus.cc:2726] T a8787bc4d6b44649b9138d906730110a P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:26.792835 29773 leader_election.cc:302] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:26.798458 30677 raft_consensus.cc:2781] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:01:26.769629 30673 consensus_queue.cc:237] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:26.807664 29773 leader_election.cc:302] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:26.794523 30614 raft_consensus.cc:686] T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d [term 2 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:26.812072 30707 raft_consensus.cc:2781] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:26.794176 30613 consensus_queue.cc:1046] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
I20240501 20:01:26.817512 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e49dd5180f04313ab33942fb3ddc147" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
W20240501 20:01:26.825551 29606 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet ece8d4ed84c34b748f7b6272b31c90ac, TS 4c8da1d580614449a636b855bf11b95f -> TS 888d99522e4844118f8f1cf26f268c8c move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:26.825968 29606 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet ece8d4ed84c34b748f7b6272b31c90ac, TS 4c8da1d580614449a636b855bf11b95f -> TS 888d99522e4844118f8f1cf26f268c8c move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:26.816581 30613 raft_consensus.cc:3032] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:26.829095 30613 raft_consensus.cc:729] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:26.817884 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e49dd5180f04313ab33942fb3ddc147" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:26.830137 29807 raft_consensus.cc:2454] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 2.
I20240501 20:01:26.829862 30613 consensus_queue.cc:260] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:26.831195 30613 raft_consensus.cc:3037] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:26.831813 29726 raft_consensus.cc:2379] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 2: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:26.833379 29625 leader_election.cc:302] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:26.815992 30713 leader_election.cc:288] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:26.805155 30677 raft_consensus.cc:686] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [term 2 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:26.847952 30677 consensus_queue.cc:237] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:26.856082 29606 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 69915dd5992641e3a24f7f45f1cd5c0c, TS fdfeac4190e045f290e003b2298d988d -> TS 888d99522e4844118f8f1cf26f268c8c move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } }
W20240501 20:01:26.856544 29606 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 69915dd5992641e3a24f7f45f1cd5c0c, TS fdfeac4190e045f290e003b2298d988d -> TS 888d99522e4844118f8f1cf26f268c8c move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } }
I20240501 20:01:26.839913 30713 raft_consensus.cc:2781] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:26.814756 30707 raft_consensus.cc:482] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:26.852497 30706 raft_consensus.cc:482] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240501 20:01:26.854768 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 47739.8us
W20240501 20:01:26.874898 29606 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 9f261d2a05bf4c838532049601694b4a, TS ee0d97247d394a8facfa774f15f3a67d -> TS 888d99522e4844118f8f1cf26f268c8c move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:26.859228 30713 raft_consensus.cc:482] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
W20240501 20:01:26.875854 29606 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 9f261d2a05bf4c838532049601694b4a, TS ee0d97247d394a8facfa774f15f3a67d -> TS 888d99522e4844118f8f1cf26f268c8c move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:26.862874 30707 raft_consensus.cc:3037] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:26.873427 30706 raft_consensus.cc:504] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:26.893577 30706 leader_election.cc:288] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:26.878551 30614 consensus_queue.cc:237] T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:26.879222 30713 raft_consensus.cc:3037] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:26.904379 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e89c0572e4ee7be8e5315b459bdb3" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:26.909183 29807 raft_consensus.cc:2454] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 2.
I20240501 20:01:26.910514 30418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e89c0572e4ee7be8e5315b459bdb3" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c" is_pre_election: true
I20240501 20:01:26.911425 30418 raft_consensus.cc:2454] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 2.
I20240501 20:01:26.912483 29625 leader_election.cc:302] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
W20240501 20:01:26.917518 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 45191us
I20240501 20:01:26.916400 30654 raft_consensus.cc:2781] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:26.928416 30654 raft_consensus.cc:482] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:26.928884 30654 raft_consensus.cc:3037] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:26.943460 30688 tablet_bootstrap.cc:492] T be383be783234855878cb8fd442c7b05 P 888d99522e4844118f8f1cf26f268c8c: Bootstrap starting.
I20240501 20:01:26.981316 30650 raft_consensus.cc:482] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:26.981930 30650 raft_consensus.cc:504] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:26.986836 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3f29f0eeea4b119984634202e5d9cd" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:26.987928 29726 raft_consensus.cc:2379] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 2: Already voted for candidate ee0d97247d394a8facfa774f15f3a67d in this term.
W20240501 20:01:26.990401 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 72241.3us
I20240501 20:01:26.983873 30650 leader_election.cc:288] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:26.991264 29556 catalog_manager.cc:5484] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 1 to 2, leader changed from fdfeac4190e045f290e003b2298d988d (127.27.217.194) to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 2 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:26.994807 29773 leader_election.cc:302] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:26.985352 30646 raft_consensus.cc:482] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:26.993570 29554 catalog_manager.cc:5484] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d reported cstate change: config changed from index 2 to -1, term changed from 1 to 2, leader changed from 4c8da1d580614449a636b855bf11b95f (127.27.217.193) to fdfeac4190e045f290e003b2298d988d (127.27.217.194), NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) evicted, now has a pending config: VOTER 4c8da1d580614449a636b855bf11b95f (127.27.217.193), VOTER fdfeac4190e045f290e003b2298d988d (127.27.217.194), VOTER ee0d97247d394a8facfa774f15f3a67d (127.27.217.195), NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196). New cstate: current_term: 2 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } } pending_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
I20240501 20:01:27.004413 29773 leader_election.cc:302] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:27.007612 29773 leader_election.cc:302] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:27.017053 29554 catalog_manager.cc:5484] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 1 to 2, leader changed from 4c8da1d580614449a636b855bf11b95f (127.27.217.193) to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 2 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:26.999228 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3f29f0eeea4b119984634202e5d9cd" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:27.016029 30697 raft_consensus.cc:482] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:27.032325 29625 leader_election.cc:302] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:27.031832 30697 raft_consensus.cc:504] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:27.034386 30650 raft_consensus.cc:3037] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:27.011509 30662 raft_consensus.cc:2781] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:01:27.013762 30677 raft_consensus.cc:2781] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:27.037930 29554 catalog_manager.cc:5484] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 1 to 2, leader changed from 4c8da1d580614449a636b855bf11b95f (127.27.217.193) to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 2 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:27.042016 29554 catalog_manager.cc:5484] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 1 to 2, leader changed from 4c8da1d580614449a636b855bf11b95f (127.27.217.193) to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 2 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:27.043233 30662 raft_consensus.cc:686] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d [term 2 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:27.044853 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ed27033ab1645e9ab0600ab635d9589" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:27.045720 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ed27033ab1645e9ab0600ab635d9589" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:27.053814 29656 raft_consensus.cc:2454] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 888d99522e4844118f8f1cf26f268c8c in term 2.
I20240501 20:01:27.046700 30707 raft_consensus.cc:504] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.015148 30646 raft_consensus.cc:504] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.049031 30713 raft_consensus.cc:504] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.043679 30697 leader_election.cc:288] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:27.062711 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac2171683884f6f97cdfeb0a656cb49" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:27.063072 30650 raft_consensus.cc:2726] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:27.063591 29726 raft_consensus.cc:2379] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 2: Already voted for candidate ee0d97247d394a8facfa774f15f3a67d in this term.
I20240501 20:01:27.064491 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e49dd5180f04313ab33942fb3ddc147" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:27.065209 29732 raft_consensus.cc:2379] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 2: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:27.065443 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f261d2a05bf4c838532049601694b4a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:27.066322 29656 raft_consensus.cc:2379] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 2: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:27.057289 29554 catalog_manager.cc:5484] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 1 to 2, leader changed from 4c8da1d580614449a636b855bf11b95f (127.27.217.193) to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 2 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:27.059142 30642 raft_consensus.cc:2781] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:01:27.037658 30677 raft_consensus.cc:482] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:27.054065 30662 consensus_queue.cc:237] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:27.052466 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 45037.6us
I20240501 20:01:27.071408 29773 leader_election.cc:302] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:27.071993 30642 raft_consensus.cc:686] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [term 2 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:27.074275 30418 raft_consensus.cc:1261] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [term 2 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 20:01:27.075314 30613 raft_consensus.cc:2781] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:27.075716 30613 raft_consensus.cc:482] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:27.076205 30646 leader_election.cc:288] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:27.070931 30677 raft_consensus.cc:3037] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:27.076398 30682 consensus_queue.cc:1035] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:01:27.077134 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac2171683884f6f97cdfeb0a656cb49" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:27.074370 30642 consensus_queue.cc:237] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.078553 29625 leader_election.cc:302] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:27.079593 30646 raft_consensus.cc:3037] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:27.081518 30707 leader_election.cc:288] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:27.082826 30383 leader_election.cc:302] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, 888d99522e4844118f8f1cf26f268c8c; no voters: fdfeac4190e045f290e003b2298d988d
I20240501 20:01:27.081056 29773 leader_election.cc:302] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:27.087335 29773 leader_election.cc:302] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:27.080921 30613 raft_consensus.cc:3037] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:27.098965 29660 raft_consensus.cc:1261] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 20:01:27.091827 29657 raft_consensus.cc:1261] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240501 20:01:27.089882 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e49dd5180f04313ab33942fb3ddc147" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:27.086179 30713 leader_election.cc:288] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:27.089300 29773 leader_election.cc:302] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:27.114163 29773 leader_election.cc:302] T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:27.108922 30707 raft_consensus.cc:2781] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:27.109694 30614 raft_consensus.cc:2781] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:27.124891 29773 leader_election.cc:302] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:27.125406 30707 raft_consensus.cc:482] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:27.130363 30707 raft_consensus.cc:3037] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:27.124718 30645 consensus_queue.cc:1035] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20240501 20:01:27.112602 30682 consensus_queue.cc:1035] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.015s
I20240501 20:01:27.126973 30614 raft_consensus.cc:482] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:27.129962 30673 raft_consensus.cc:2781] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:27.123720 30671 raft_consensus.cc:2781] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:01:27.141752 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f261d2a05bf4c838532049601694b4a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:27.139189 30740 raft_consensus.cc:2781] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:27.132145 30642 raft_consensus.cc:2781] T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:27.137141 30614 raft_consensus.cc:3037] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:27.137115 30673 raft_consensus.cc:482] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:27.137120 30678 consensus_queue.cc:1035] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
I20240501 20:01:27.131546 30654 raft_consensus.cc:504] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:27.152061 30673 raft_consensus.cc:3037] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:27.176112 29726 raft_consensus.cc:2454] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 2.
I20240501 20:01:27.148769 30642 raft_consensus.cc:482] T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:27.186244 30687 raft_consensus.cc:482] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:27.186964 29807 raft_consensus.cc:2454] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 2.
I20240501 20:01:27.186789 30687 raft_consensus.cc:504] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.175014 30697 raft_consensus.cc:482] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:27.188730 30687 leader_election.cc:288] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:27.188295 30697 raft_consensus.cc:504] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:27.190609 30697 leader_election.cc:288] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:27.190529 30707 raft_consensus.cc:504] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.182922 30642 raft_consensus.cc:3037] T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:27.192901 29805 raft_consensus.cc:3037] T ec073fe645614fe28e7837de3c22510b P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:27.193846 30654 leader_election.cc:288] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Requested vote from peers ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:27.195008 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd5e83faaff4e2da4e4cc22ba410d33" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
W20240501 20:01:27.196626 30739 log.cc:924] Time spent T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20240501 20:01:27.197372 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd5e83faaff4e2da4e4cc22ba410d33" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:27.198156 30677 raft_consensus.cc:504] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.198582 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649b71f9f267437ca51c758dd9ca6987" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:27.199399 29653 raft_consensus.cc:2454] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 2.
I20240501 20:01:27.200898 29772 leader_election.cc:302] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:27.201983 30662 raft_consensus.cc:2781] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:27.202347 30662 raft_consensus.cc:482] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:27.202734 30662 raft_consensus.cc:3037] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:27.203517 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e38a2e91a3d48d09a22abce723af833" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:27.204335 29656 raft_consensus.cc:2379] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 2: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:27.204581 30707 leader_election.cc:288] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:27.198418 30646 raft_consensus.cc:2726] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:27.210757 30383 leader_election.cc:302] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 888d99522e4844118f8f1cf26f268c8c; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:27.211992 30697 raft_consensus.cc:2726] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240501 20:01:27.214587 30642 raft_consensus.cc:504] T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.215513 29807 raft_consensus.cc:3037] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:27.216688 30642 leader_election.cc:288] T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:27.218892 30677 leader_election.cc:288] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:27.222038 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e49dd5180f04313ab33942fb3ddc147" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:27.222757 29727 raft_consensus.cc:2454] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 2.
I20240501 20:01:27.226269 30740 raft_consensus.cc:482] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [term 2 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:27.226927 30740 raft_consensus.cc:3037] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:27.229511 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458740ab5c444b6ab3d1991c69db5084" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:27.208979 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8edcd903224d48f889b46e1b5318e0ec" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:27.231725 29656 raft_consensus.cc:2379] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 2: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:27.233067 29653 raft_consensus.cc:2379] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 2: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:27.226614 30418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e89c0572e4ee7be8e5315b459bdb3" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c"
I20240501 20:01:27.241961 30697 raft_consensus.cc:482] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:27.245275 30419 raft_consensus.cc:1261] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [term 4 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240501 20:01:27.246114 30678 raft_consensus.cc:482] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:27.245906 30697 raft_consensus.cc:504] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:27.246611 30678 raft_consensus.cc:504] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.248530 30678 leader_election.cc:288] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:27.249727 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e07220db7444ae97f294d70eda6a93" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:27.250485 29653 raft_consensus.cc:2454] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 888d99522e4844118f8f1cf26f268c8c in term 2.
I20240501 20:01:27.251461 30383 leader_election.cc:302] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, 888d99522e4844118f8f1cf26f268c8c; no voters:
I20240501 20:01:27.240990 29556 catalog_manager.cc:5484] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 1 to 2, leader changed from fdfeac4190e045f290e003b2298d988d (127.27.217.194) to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 2 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:27.254586 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:27.255265 29659 raft_consensus.cc:2454] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
I20240501 20:01:27.256830 29556 catalog_manager.cc:5484] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 1 to 2, leader changed from fdfeac4190e045f290e003b2298d988d (127.27.217.194) to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 2 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:27.263276 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458740ab5c444b6ab3d1991c69db5084" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:27.265077 29731 raft_consensus.cc:1261] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:01:27.265475 29625 leader_election.cc:302] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters: fdfeac4190e045f290e003b2298d988d
I20240501 20:01:27.265842 29659 raft_consensus.cc:1261] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:01:27.267174 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:27.267608 29730 raft_consensus.cc:2454] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 2.
I20240501 20:01:27.262408 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e38a2e91a3d48d09a22abce723af833" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
W20240501 20:01:27.268836 29606 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet be383be783234855878cb8fd442c7b05, TS fdfeac4190e045f290e003b2298d988d -> TS 888d99522e4844118f8f1cf26f268c8c move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:27.331367 29772 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 66697.2us
W20240501 20:01:27.331426 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 73967.5us
I20240501 20:01:27.332087 29656 raft_consensus.cc:3037] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:27.238992 30671 raft_consensus.cc:686] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [term 2 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
W20240501 20:01:27.333544 29698 consensus_peers.cc:485] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d -> Peer 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227): Couldn't send request to peer 888d99522e4844118f8f1cf26f268c8c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20240501 20:01:27.331830 29606 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet be383be783234855878cb8fd442c7b05, TS fdfeac4190e045f290e003b2298d988d -> TS 888d99522e4844118f8f1cf26f268c8c move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.331375 29556 catalog_manager.cc:5484] T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 1 to 2, leader changed from fdfeac4190e045f290e003b2298d988d (127.27.217.194) to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 2 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:27.203492 30614 raft_consensus.cc:504] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.345562 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8edcd903224d48f889b46e1b5318e0ec" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:27.345242 29556 catalog_manager.cc:5484] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 1 to 2, leader changed from fdfeac4190e045f290e003b2298d988d (127.27.217.194) to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 2 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:27.346786 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649b71f9f267437ca51c758dd9ca6987" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:27.347510 29728 raft_consensus.cc:2454] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 2.
I20240501 20:01:27.350982 29733 raft_consensus.cc:2454] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 2.
I20240501 20:01:27.264964 30418 raft_consensus.cc:3037] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 2 FOLLOWER]: Advancing to term 3
W20240501 20:01:27.344445 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.196:37227, user_credentials={real_user=slave}} blocked reactor thread for 80934.5us
I20240501 20:01:27.337168 30650 raft_consensus.cc:2781] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:01:27.331182 30744 raft_consensus.cc:482] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:27.343811 30643 raft_consensus.cc:482] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:27.380084 30646 raft_consensus.cc:482] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:27.380663 30646 raft_consensus.cc:504] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.382730 30646 leader_election.cc:288] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:27.388211 30614 leader_election.cc:288] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:27.388464 30740 raft_consensus.cc:504] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:27.390806 30740 leader_election.cc:288] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 3 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:27.391397 30740 raft_consensus.cc:2781] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:27.390825 29732 raft_consensus.cc:2454] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 2.
I20240501 20:01:27.393162 30662 raft_consensus.cc:504] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.395627 30662 leader_election.cc:288] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:27.393688 29656 raft_consensus.cc:1261] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:01:27.391768 30740 raft_consensus.cc:482] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:27.383755 29735 raft_consensus.cc:2454] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 2.
I20240501 20:01:27.362099 30650 raft_consensus.cc:686] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [term 2 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:27.358554 30697 leader_election.cc:288] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:27.398597 30744 raft_consensus.cc:504] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.441399 30744 leader_election.cc:288] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:27.442272 30650 consensus_queue.cc:237] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.444155 30671 consensus_queue.cc:237] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.448621 30649 raft_consensus.cc:482] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:27.449359 30649 raft_consensus.cc:504] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.450997 29805 raft_consensus.cc:1261] T ec073fe645614fe28e7837de3c22510b P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:01:27.400400 30643 raft_consensus.cc:504] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.453647 30649 leader_election.cc:288] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:27.408380 30645 consensus_queue.cc:1035] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.046s
I20240501 20:01:27.454603 30643 leader_election.cc:288] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:27.399365 30740 raft_consensus.cc:3037] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:27.466696 30613 raft_consensus.cc:504] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.467625 30673 raft_consensus.cc:504] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.450119 29807 raft_consensus.cc:1261] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 20:01:27.491850 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f261d2a05bf4c838532049601694b4a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:27.496675 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c597f15e844f30837012baed4b0501" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:27.497360 29733 raft_consensus.cc:2454] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:27.492781 29735 raft_consensus.cc:2454] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 2.
I20240501 20:01:27.499272 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a2bba62f434d1c8489b55c8fec2187" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:27.499950 29732 raft_consensus.cc:2454] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 2.
I20240501 20:01:27.483275 30685 raft_consensus.cc:482] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:27.503361 30744 raft_consensus.cc:482] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:27.496934 30703 consensus_queue.cc:1035] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20240501 20:01:27.543602 29624 leader_election.cc:302] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:27.544533 30649 raft_consensus.cc:2781] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:27.544937 30649 raft_consensus.cc:482] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:27.545387 30649 raft_consensus.cc:3037] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:27.545004 30703 consensus_queue.cc:1035] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 20:01:27.516476 30685 raft_consensus.cc:504] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.550628 30685 leader_election.cc:288] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:27.552700 29624 leader_election.cc:302] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:27.554234 30643 raft_consensus.cc:2781] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:27.554694 30643 raft_consensus.cc:482] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:27.555111 30643 raft_consensus.cc:3037] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:27.555498 30418 raft_consensus.cc:2454] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 3.
I20240501 20:01:27.557147 29623 leader_election.cc:302] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, 888d99522e4844118f8f1cf26f268c8c; no voters:
I20240501 20:01:27.558223 30706 raft_consensus.cc:2781] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:01:27.558851 30706 raft_consensus.cc:686] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:27.559949 30706 consensus_queue.cc:237] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:27.518700 30744 raft_consensus.cc:504] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.564836 30649 raft_consensus.cc:504] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.566601 30744 leader_election.cc:288] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:27.573609 30613 leader_election.cc:288] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:27.539155 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e07220db7444ae97f294d70eda6a93" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:27.540442 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ed27033ab1645e9ab0600ab635d9589" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:27.574604 30643 raft_consensus.cc:504] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.543390 30673 leader_election.cc:288] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:27.588550 29624 leader_election.cc:302] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:27.590260 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f261d2a05bf4c838532049601694b4a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:27.590590 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 253380us
I20240501 20:01:27.582317 30740 raft_consensus.cc:504] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:27.591553 30752 raft_consensus.cc:482] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:27.598299 30654 raft_consensus.cc:2781] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:27.598178 30752 raft_consensus.cc:504] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.598845 30654 raft_consensus.cc:482] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:27.599332 30654 raft_consensus.cc:3037] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:27.596215 29773 leader_election.cc:302] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:27.607414 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e07220db7444ae97f294d70eda6a93" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:27.610616 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c597f15e844f30837012baed4b0501" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:27.611382 29733 raft_consensus.cc:3037] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 3
I20240501 20:01:27.623855 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c597f15e844f30837012baed4b0501" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:27.626226 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a2bba62f434d1c8489b55c8fec2187" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:27.627219 29803 raft_consensus.cc:2454] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 2.
I20240501 20:01:27.624745 29809 raft_consensus.cc:2454] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 2.
W20240501 20:01:27.628567 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.196:37227, user_credentials={real_user=slave}} blocked reactor thread for 270600us
I20240501 20:01:27.629567 29773 leader_election.cc:302] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:27.633096 29657 raft_consensus.cc:1261] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 20:01:27.628633 30753 raft_consensus.cc:482] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:27.629843 30673 raft_consensus.cc:2781] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:01:27.634589 29772 leader_election.cc:302] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:27.636767 29657 raft_consensus.cc:1261] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 20:01:27.638793 29660 raft_consensus.cc:1261] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:01:27.600337 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5525f546426c47a38edf7f56ad04b7ab" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:27.600626 30748 raft_consensus.cc:482] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:27.617739 30740 leader_election.cc:288] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 3 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:27.617688 30649 leader_election.cc:288] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:27.618018 30646 raft_consensus.cc:482] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:27.644883 30713 raft_consensus.cc:482] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:27.645212 30646 raft_consensus.cc:504] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.645378 30713 raft_consensus.cc:504] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.646983 29806 raft_consensus.cc:2454] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 2.
I20240501 20:01:27.647382 30646 leader_election.cc:288] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:27.642812 29660 raft_consensus.cc:1261] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:01:27.647648 30748 raft_consensus.cc:504] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.641423 30752 leader_election.cc:288] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:27.649811 30748 leader_election.cc:288] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:27.644136 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5525f546426c47a38edf7f56ad04b7ab" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:27.650367 30677 raft_consensus.cc:2781] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:27.650926 29657 raft_consensus.cc:2379] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d in current term 2: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:27.654907 29700 leader_election.cc:302] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:27.650810 30677 raft_consensus.cc:482] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:27.633997 30753 raft_consensus.cc:504] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.683085 30645 consensus_queue.cc:1035] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 20:01:27.683720 30677 raft_consensus.cc:3037] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:27.684753 30688 tablet_bootstrap.cc:492] T be383be783234855878cb8fd442c7b05 P 888d99522e4844118f8f1cf26f268c8c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:01:27.685379 29806 raft_consensus.cc:1261] T 456035a1cb3f4f7da52d0fc1dd1c656c P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:01:27.685703 30688 tablet_bootstrap.cc:492] T be383be783234855878cb8fd442c7b05 P 888d99522e4844118f8f1cf26f268c8c: Bootstrap complete.
I20240501 20:01:27.687098 30418 raft_consensus.cc:1226] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 2. Current term is 3. Ops: []
I20240501 20:01:27.687232 30688 ts_tablet_manager.cc:1367] T be383be783234855878cb8fd442c7b05 P 888d99522e4844118f8f1cf26f268c8c: Time spent bootstrapping tablet: real 0.744s user 0.021s sys 0.029s
I20240501 20:01:27.687186 30419 raft_consensus.cc:1226] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 2. Current term is 3. Ops: []
I20240501 20:01:27.645129 30643 leader_election.cc:288] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:27.647397 30713 leader_election.cc:288] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:27.690646 30654 raft_consensus.cc:504] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.690573 30688 raft_consensus.cc:348] T be383be783234855878cb8fd442c7b05 P 888d99522e4844118f8f1cf26f268c8c [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:27.691565 30688 raft_consensus.cc:729] T be383be783234855878cb8fd442c7b05 P 888d99522e4844118f8f1cf26f268c8c [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 888d99522e4844118f8f1cf26f268c8c, State: Initialized, Role: LEARNER
I20240501 20:01:27.692262 30688 consensus_queue.cc:260] T be383be783234855878cb8fd442c7b05 P 888d99522e4844118f8f1cf26f268c8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:27.693622 30756 raft_consensus.cc:482] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:27.694093 30756 raft_consensus.cc:504] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.694412 30677 raft_consensus.cc:504] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.696130 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c597f15e844f30837012baed4b0501" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:27.696664 30761 raft_consensus.cc:2781] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:27.697306 30761 raft_consensus.cc:482] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:27.697775 30761 raft_consensus.cc:3037] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:27.698582 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e89c0572e4ee7be8e5315b459bdb3" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:27.699187 29805 raft_consensus.cc:3037] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:27.696815 29809 raft_consensus.cc:3037] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:27.702327 30753 leader_election.cc:288] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:27.703275 30654 leader_election.cc:288] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:27.709506 29733 raft_consensus.cc:2454] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 3.
W20240501 20:01:27.710003 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 66295.4us
I20240501 20:01:27.708523 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31fff64b453a423eb0f567b3171564fb" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:27.711097 29808 raft_consensus.cc:2454] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
W20240501 20:01:27.716468 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 62119.4us
I20240501 20:01:27.717001 29700 leader_election.cc:302] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:27.723711 29809 raft_consensus.cc:2454] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 3.
I20240501 20:01:27.724458 30673 raft_consensus.cc:686] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [term 2 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:27.713905 30755 raft_consensus.cc:2781] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:27.726190 30755 raft_consensus.cc:482] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:27.724537 30682 consensus_queue.cc:1035] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20240501 20:01:27.726581 30673 consensus_queue.cc:237] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.728003 30762 consensus_queue.cc:1035] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240501 20:01:27.729394 30762 consensus_queue.cc:1035] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:27.729907 30682 consensus_queue.cc:1035] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:27.737334 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f5d1686214485d91a0baf4b870740e" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:27.738166 29804 raft_consensus.cc:2454] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 2.
I20240501 20:01:27.741529 29806 raft_consensus.cc:3037] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:27.740981 30751 raft_consensus.cc:482] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:27.743436 30751 raft_consensus.cc:504] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.745397 30751 leader_election.cc:288] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:27.726789 30703 raft_consensus.cc:2781] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:27.747138 30703 raft_consensus.cc:482] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:27.747501 30703 raft_consensus.cc:3037] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:27.715909 30743 consensus_queue.cc:1035] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20240501 20:01:27.738782 30756 leader_election.cc:288] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:27.748848 29773 leader_election.cc:302] T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:27.726547 30755 raft_consensus.cc:3037] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:27.750640 29700 leader_election.cc:302] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:27.751543 29773 leader_election.cc:302] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:27.752564 30751 raft_consensus.cc:2781] T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:01:27.755050 30751 raft_consensus.cc:686] T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d [term 2 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:27.755962 30751 consensus_queue.cc:237] T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.756974 30753 raft_consensus.cc:2781] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:27.757359 30753 raft_consensus.cc:482] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:27.757766 30753 raft_consensus.cc:3037] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:27.758833 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07bdb8b09b8c4ef4b8e49daa9b8c01d6" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:27.759469 29808 raft_consensus.cc:2454] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 2.
I20240501 20:01:27.760850 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:27.761004 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece8d4ed84c34b748f7b6272b31c90ac" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:27.760998 29625 leader_election.cc:302] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:27.761749 29803 raft_consensus.cc:2454] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 2.
I20240501 20:01:27.762176 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a2bba62f434d1c8489b55c8fec2187" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:27.762737 29800 raft_consensus.cc:3037] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:27.764796 30713 raft_consensus.cc:2781] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:01:27.765264 30713 raft_consensus.cc:686] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
W20240501 20:01:27.765723 30383 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.513s user 0.009s sys 0.015s
I20240501 20:01:27.766028 30713 consensus_queue.cc:237] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.761428 29808 raft_consensus.cc:2454] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 2.
I20240501 20:01:27.769431 29800 raft_consensus.cc:2454] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 3.
I20240501 20:01:27.770941 29625 leader_election.cc:302] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:27.771801 30650 raft_consensus.cc:2781] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:27.771914 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f261d2a05bf4c838532049601694b4a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:27.772320 30650 raft_consensus.cc:482] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:27.772691 30650 raft_consensus.cc:3037] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:27.775199 29625 leader_election.cc:302] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:27.751724 30743 consensus_queue.cc:1035] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20240501 20:01:27.775997 30748 raft_consensus.cc:2781] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:27.776662 30748 raft_consensus.cc:482] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:27.776993 30748 raft_consensus.cc:3037] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:27.780202 29625 leader_election.cc:302] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:27.781260 29773 leader_election.cc:302] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:27.781883 29625 leader_election.cc:302] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:27.779206 29556 catalog_manager.cc:5484] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 1 to 2, leader changed from fdfeac4190e045f290e003b2298d988d (127.27.217.194) to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 2 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:27.783224 30646 raft_consensus.cc:2781] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:01:27.783761 30646 raft_consensus.cc:686] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:27.783991 29556 catalog_manager.cc:5484] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 1 to 2, leader changed from fdfeac4190e045f290e003b2298d988d (127.27.217.194) to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 2 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:27.784612 30646 consensus_queue.cc:237] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.785573 29556 catalog_manager.cc:5484] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 1 to 2, leader changed from fdfeac4190e045f290e003b2298d988d (127.27.217.194) to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 2 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:27.771351 29557 catalog_manager.cc:5484] T be383be783234855878cb8fd442c7b05 P 888d99522e4844118f8f1cf26f268c8c reported cstate change: config changed from index -1 to 2, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added, no longer has a pending config: VOTER 4c8da1d580614449a636b855bf11b95f (127.27.217.193), VOTER fdfeac4190e045f290e003b2298d988d (127.27.217.194), VOTER ee0d97247d394a8facfa774f15f3a67d (127.27.217.195), NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196). New cstate: current_term: 2 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
W20240501 20:01:27.766062 30383 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.514s user 0.009s sys 0.015s
I20240501 20:01:27.759270 30613 raft_consensus.cc:2781] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:01:27.791603 30613 raft_consensus.cc:686] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [term 2 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
W20240501 20:01:27.792001 30383 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 540603us
I20240501 20:01:27.775486 30688 ts_tablet_manager.cc:1398] T be383be783234855878cb8fd442c7b05 P 888d99522e4844118f8f1cf26f268c8c: Time spent starting tablet: real 0.088s user 0.008s sys 0.000s
I20240501 20:01:27.796576 30613 consensus_queue.cc:237] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.797197 30754 raft_consensus.cc:2781] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:01:27.768631 29805 raft_consensus.cc:2454] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 3.
I20240501 20:01:27.788729 29554 catalog_manager.cc:5484] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 1 to 2, leader changed from ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 2 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:27.794111 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ed27033ab1645e9ab0600ab635d9589" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:27.789629 30643 raft_consensus.cc:2781] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:27.797690 30754 raft_consensus.cc:686] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d [term 2 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:27.789727 30677 leader_election.cc:288] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:27.810846 30706 raft_consensus.cc:482] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:27.808131 30643 raft_consensus.cc:482] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:27.814385 30706 raft_consensus.cc:504] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.814944 30643 raft_consensus.cc:3037] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:27.816345 30706 leader_election.cc:288] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:27.817664 30755 raft_consensus.cc:504] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.818338 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5525fc4b82a44b0991c3dd6098fde81" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:27.814973 30754 consensus_queue.cc:237] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.804906 30654 raft_consensus.cc:482] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:27.838106 30748 raft_consensus.cc:504] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.841442 30706 raft_consensus.cc:482] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:27.841966 30706 raft_consensus.cc:504] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.843082 30748 leader_election.cc:288] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:27.843930 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece8d4ed84c34b748f7b6272b31c90ac" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:27.844511 29800 raft_consensus.cc:3037] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:27.844089 30643 raft_consensus.cc:504] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.846477 30643 leader_election.cc:288] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:27.851197 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49da4246f5f6438abe0e7e8c9a523d80" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:27.831027 29805 raft_consensus.cc:2454] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 2.
I20240501 20:01:27.852442 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:27.853188 29808 raft_consensus.cc:3037] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:27.853372 30761 raft_consensus.cc:504] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.855935 30761 leader_election.cc:288] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:27.851871 29804 raft_consensus.cc:2454] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 2.
I20240501 20:01:27.856472 29625 leader_election.cc:302] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:27.858491 30649 raft_consensus.cc:2781] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:27.858857 30649 raft_consensus.cc:482] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:27.859206 30649 raft_consensus.cc:3037] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:27.860284 29625 leader_election.cc:302] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:27.864691 29806 raft_consensus.cc:1261] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:01:27.864908 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5525f546426c47a38edf7f56ad04b7ab" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:27.865828 30643 raft_consensus.cc:2781] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:27.866235 30643 raft_consensus.cc:482] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:27.866264 30761 consensus_queue.cc:1035] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 20:01:27.843809 30706 leader_election.cc:288] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:27.872617 29808 raft_consensus.cc:2454] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 3.
I20240501 20:01:27.831329 30755 leader_election.cc:288] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:27.874240 29625 leader_election.cc:302] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:27.885706 29804 raft_consensus.cc:2454] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 2.
I20240501 20:01:27.887094 30650 raft_consensus.cc:504] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.888120 29700 leader_election.cc:302] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:27.889369 30761 raft_consensus.cc:2781] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:01:27.889711 30650 leader_election.cc:288] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:27.889972 30761 raft_consensus.cc:686] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 2 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:27.890465 30650 raft_consensus.cc:2781] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:01:27.873523 30706 raft_consensus.cc:482] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:27.891348 30761 consensus_queue.cc:237] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.892153 30706 raft_consensus.cc:504] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.894816 30706 leader_election.cc:288] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:27.863899 29800 raft_consensus.cc:2454] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 3.
I20240501 20:01:27.897707 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458740ab5c444b6ab3d1991c69db5084" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:27.899199 29625 leader_election.cc:302] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:27.902558 30706 raft_consensus.cc:2781] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:01:27.899719 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07bdb8b09b8c4ef4b8e49daa9b8c01d6" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:27.904022 29808 raft_consensus.cc:3037] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:27.866712 30643 raft_consensus.cc:3037] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:27.890981 30650 raft_consensus.cc:686] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:27.864367 30703 raft_consensus.cc:504] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.903198 30706 raft_consensus.cc:686] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:27.832268 30654 raft_consensus.cc:504] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:27.924515 30762 raft_consensus.cc:482] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:27.943197 30642 raft_consensus.cc:482] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:27.959270 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07bdb8b09b8c4ef4b8e49daa9b8c01d6" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:27.960211 29733 raft_consensus.cc:2454] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 2.
I20240501 20:01:27.964790 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece8d4ed84c34b748f7b6272b31c90ac" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:27.965642 29733 raft_consensus.cc:2454] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 2.
I20240501 20:01:27.971035 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a2bba62f434d1c8489b55c8fec2187" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:27.971617 29735 raft_consensus.cc:3037] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:27.972267 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f261d2a05bf4c838532049601694b4a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:27.972430 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:27.972874 29728 raft_consensus.cc:3037] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:27.973186 29729 raft_consensus.cc:2454] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:27.944911 30769 raft_consensus.cc:482] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:27.944592 30748 raft_consensus.cc:482] T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:27.992357 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5525f546426c47a38edf7f56ad04b7ab" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:27.993409 29805 raft_consensus.cc:2454] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 2.
I20240501 20:01:27.995092 29625 leader_election.cc:302] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:27.996243 30671 raft_consensus.cc:482] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:27.996922 30671 raft_consensus.cc:504] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:28.001197 30671 leader_election.cc:288] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:28.002077 30419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05df43bdfa479b902b1da297fb5974" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c" is_pre_election: true
I20240501 20:01:28.003206 30419 raft_consensus.cc:2379] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 2: Already voted for candidate 888d99522e4844118f8f1cf26f268c8c in this term.
I20240501 20:01:28.005548 30654 leader_election.cc:288] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:28.007297 30654 raft_consensus.cc:482] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:28.050395 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece8d4ed84c34b748f7b6272b31c90ac" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:28.051127 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49da4246f5f6438abe0e7e8c9a523d80" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:28.051203 29733 raft_consensus.cc:3037] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:28.064496 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458740ab5c444b6ab3d1991c69db5084" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:28.065402 29730 raft_consensus.cc:2454] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 2.
W20240501 20:01:28.048365 30765 log.cc:924] Time spent T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 0.145s user 0.001s sys 0.001s
I20240501 20:01:28.075884 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:28.076944 29734 raft_consensus.cc:3037] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 3
I20240501 20:01:27.942800 30650 consensus_queue.cc:237] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:28.080952 29735 raft_consensus.cc:2454] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 3.
I20240501 20:01:28.089354 29733 raft_consensus.cc:2454] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 3.
I20240501 20:01:28.081872 29808 raft_consensus.cc:2454] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 3.
I20240501 20:01:28.007869 30654 raft_consensus.cc:504] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:28.103821 30654 leader_election.cc:288] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:28.080451 30760 raft_consensus.cc:482] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:28.108839 29728 raft_consensus.cc:2454] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 3.
I20240501 20:01:28.110189 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07bdb8b09b8c4ef4b8e49daa9b8c01d6" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:28.112071 30706 consensus_queue.cc:237] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:28.048789 30748 raft_consensus.cc:504] T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:28.093175 30753 raft_consensus.cc:504] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:28.113535 30703 leader_election.cc:288] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:28.124135 29734 raft_consensus.cc:2454] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 3.
I20240501 20:01:27.977401 30642 raft_consensus.cc:504] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:28.126039 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31fff64b453a423eb0f567b3171564fb" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:27.981094 30769 raft_consensus.cc:504] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:28.127385 29731 raft_consensus.cc:3037] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:28.092765 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5525fc4b82a44b0991c3dd6098fde81" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:27.979647 30762 raft_consensus.cc:504] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:28.150142 29804 raft_consensus.cc:3037] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 3
I20240501 20:01:28.128523 30760 raft_consensus.cc:504] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:28.171272 29731 raft_consensus.cc:2454] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 3.
I20240501 20:01:28.172755 30760 leader_election.cc:288] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:28.165261 30654 raft_consensus.cc:2781] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:28.177263 29729 raft_consensus.cc:2454] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:28.180435 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "696121417ea9403990456cfc6fc4f416" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:28.181793 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5525fc4b82a44b0991c3dd6098fde81" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:28.159035 30748 leader_election.cc:288] T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:28.180640 30762 leader_election.cc:288] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:28.210412 29808 raft_consensus.cc:2379] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d in current term 2: Already voted for candidate ee0d97247d394a8facfa774f15f3a67d in this term.
I20240501 20:01:28.194794 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "696121417ea9403990456cfc6fc4f416" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:28.217785 30703 raft_consensus.cc:482] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:28.220177 30759 raft_consensus.cc:482] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
W20240501 20:01:28.232676 29698 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.507s user 0.023s sys 0.012s
W20240501 20:01:28.233002 29698 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.508s user 0.023s sys 0.012s
W20240501 20:01:28.233268 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 508214us
W20240501 20:01:28.235793 30673 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.505s user 0.007s sys 0.022s
I20240501 20:01:28.212718 30654 raft_consensus.cc:482] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:28.225003 30713 raft_consensus.cc:482] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:28.232375 30748 raft_consensus.cc:482] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:28.262574 29728 raft_consensus.cc:2379] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 888d99522e4844118f8f1cf26f268c8c in current term 2: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:28.244211 30645 raft_consensus.cc:482] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:28.233798 30768 raft_consensus.cc:482] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:28.275463 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f5d1686214485d91a0baf4b870740e" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:28.308863 29808 raft_consensus.cc:3037] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:28.231618 29807 raft_consensus.cc:2454] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 888d99522e4844118f8f1cf26f268c8c in term 2.
W20240501 20:01:28.257974 30673 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.528s user 0.008s sys 0.022s
W20240501 20:01:28.256654 29802 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.530s user 0.001s sys 0.024s
I20240501 20:01:28.258587 30654 raft_consensus.cc:3037] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:28.234526 30703 raft_consensus.cc:504] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:28.319803 29804 raft_consensus.cc:2454] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 3.
I20240501 20:01:28.244877 30759 raft_consensus.cc:504] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:28.266633 30649 raft_consensus.cc:504] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:28.276211 30753 leader_election.cc:288] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:28.356753 30753 raft_consensus.cc:482] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:28.357298 30753 raft_consensus.cc:504] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:28.359177 30753 leader_election.cc:288] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:28.352218 29802 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.625s user 0.002s sys 0.024s
I20240501 20:01:28.323801 30769 leader_election.cc:288] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:28.274658 30713 raft_consensus.cc:504] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:28.321658 30642 leader_election.cc:288] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:28.367165 30649 leader_election.cc:288] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:28.316975 30645 raft_consensus.cc:504] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:28.369509 30645 leader_election.cc:288] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:28.370142 30645 raft_consensus.cc:482] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:28.370558 30645 raft_consensus.cc:504] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:28.375205 30713 leader_election.cc:288] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:28.377108 30703 leader_election.cc:288] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:28.320869 30768 raft_consensus.cc:504] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:28.255755 30685 consensus_queue.cc:1035] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.061s
I20240501 20:01:28.306242 30756 raft_consensus.cc:482] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240501 20:01:28.326984 29801 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.515s user 0.003s sys 0.017s
I20240501 20:01:28.277339 30748 raft_consensus.cc:504] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:28.382193 29801 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.571s user 0.003s sys 0.017s
W20240501 20:01:28.327807 30754 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.501s user 0.014s sys 0.004s
I20240501 20:01:28.340097 29808 raft_consensus.cc:2454] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 3.
W20240501 20:01:28.343127 30751 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.584s user 0.007s sys 0.034s
I20240501 20:01:28.323217 30643 raft_consensus.cc:504] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:28.359889 30753 raft_consensus.cc:482] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:28.388782 30748 leader_election.cc:288] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:28.394956 30654 raft_consensus.cc:504] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:28.389982 30759 leader_election.cc:288] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:28.401707 30613 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.571s user 0.021s sys 0.029s
W20240501 20:01:28.362586 30383 thread.cc:636] server-negotiator [worker] (thread pool) Time spent creating pthread: real 0.569s user 0.010s sys 0.011s
W20240501 20:01:28.401253 30751 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.642s user 0.007s sys 0.034s
W20240501 20:01:28.385735 30754 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.559s user 0.014s sys 0.005s
W20240501 20:01:28.409001 30722 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 0.529s user 0.017s sys 0.032s
W20240501 20:01:28.416658 29809 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.667s user 0.010s sys 0.014s
W20240501 20:01:28.417677 29809 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.668s user 0.010s sys 0.014s
W20240501 20:01:28.410526 30613 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.580s user 0.022s sys 0.029s
I20240501 20:01:28.385452 30756 raft_consensus.cc:504] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:28.412287 30383 thread.cc:602] server-negotiator [worker] (thread pool) Time spent starting thread: real 0.619s user 0.010s sys 0.012s
W20240501 20:01:28.443905 29625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 448904us
I20240501 20:01:28.414973 30713 raft_consensus.cc:482] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:28.464388 30713 raft_consensus.cc:504] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:28.465902 30769 raft_consensus.cc:482] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:28.466461 30713 leader_election.cc:288] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:28.466998 30713 raft_consensus.cc:482] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:28.467424 30713 raft_consensus.cc:504] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:28.467782 30646 raft_consensus.cc:482] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
W20240501 20:01:28.435117 30722 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 0.563s user 0.017s sys 0.033s
I20240501 20:01:28.479905 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5525f546426c47a38edf7f56ad04b7ab" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:28.480808 29733 raft_consensus.cc:1261] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:01:28.468941 30740 raft_consensus.cc:482] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:28.470191 30383 leader_election.cc:302] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 888d99522e4844118f8f1cf26f268c8c; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:28.470497 30649 raft_consensus.cc:482] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:28.470906 29801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcf30cf66714efdac997862dafbe4a2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:28.422389 30703 raft_consensus.cc:482] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:28.472968 30643 leader_election.cc:288] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:28.474107 30654 leader_election.cc:288] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:28.412880 30748 raft_consensus.cc:482] T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:28.403625 30753 raft_consensus.cc:504] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:28.567487 30740 raft_consensus.cc:504] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:28.594971 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5525fc4b82a44b0991c3dd6098fde81" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:28.585232 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31fff64b453a423eb0f567b3171564fb" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:28.558666 30685 raft_consensus.cc:482] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:28.528381 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e9f75387564275a60d38e9a910319c" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:28.521203 30759 raft_consensus.cc:482] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:28.576594 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8edcd903224d48f889b46e1b5318e0ec" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:28.466473 30769 raft_consensus.cc:504] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:28.550487 29734 raft_consensus.cc:1261] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:01:28.475267 30645 leader_election.cc:288] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:28.549367 30761 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.654s user 0.006s sys 0.020s
I20240501 20:01:28.567557 30682 raft_consensus.cc:482] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:28.548219 30771 raft_consensus.cc:482] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:28.568725 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0374bb790ed345ed90452595f71383a2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:28.574333 29733 raft_consensus.cc:1261] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:01:28.478942 30768 leader_election.cc:288] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:28.567396 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07bdb8b09b8c4ef4b8e49daa9b8c01d6" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:28.481465 30646 raft_consensus.cc:504] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:28.599867 29735 raft_consensus.cc:1261] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:01:28.625977 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece8d4ed84c34b748f7b6272b31c90ac" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:28.654773 29730 raft_consensus.cc:2354] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d for earlier term 2. Current term is 3.
I20240501 20:01:28.630046 29801 raft_consensus.cc:2454] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 2.
I20240501 20:01:28.655346 29735 raft_consensus.cc:1261] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:01:28.638219 29803 raft_consensus.cc:2379] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 3: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
W20240501 20:01:28.630982 30650 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.501s user 0.011s sys 0.014s
I20240501 20:01:28.599561 30756 leader_election.cc:288] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:28.641830 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a35893b91247a99912eba787d787cf" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:28.662940 29801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcf30cf66714efdac997862dafbe4a2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:28.663723 29801 raft_consensus.cc:2454] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 2.
I20240501 20:01:28.645571 29805 raft_consensus.cc:3037] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:28.660080 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49da4246f5f6438abe0e7e8c9a523d80" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:28.670182 29801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36057b3f24d44b9aac3a1378053a395f" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:28.647089 30761 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.752s user 0.009s sys 0.021s
I20240501 20:01:28.662879 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48fd0a8e289401aa7417c42ff2c81b2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:28.673247 29803 raft_consensus.cc:2379] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d in current term 4: Already voted for candidate ee0d97247d394a8facfa774f15f3a67d in this term.
I20240501 20:01:28.600741 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a35893b91247a99912eba787d787cf" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:28.659077 29732 raft_consensus.cc:2454] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 2.
I20240501 20:01:28.668035 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49da4246f5f6438abe0e7e8c9a523d80" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:28.636968 30703 raft_consensus.cc:504] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:28.668691 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b337b8e3e38d4ad6b1076fbd0d0a1609" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:28.660593 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6797d82dda2948d8a7445fcd33e44752" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:28.670779 29733 raft_consensus.cc:2454] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
W20240501 20:01:28.660529 30650 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.530s user 0.011s sys 0.014s
I20240501 20:01:28.634555 30649 raft_consensus.cc:504] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:28.725162 29800 raft_consensus.cc:2454] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 2.
I20240501 20:01:28.668370 29700 leader_election.cc:302] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:28.642596 30748 raft_consensus.cc:504] T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:28.666384 30788 raft_consensus.cc:482] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:28.741617 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e49dd5180f04313ab33942fb3ddc147" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:28.664911 30760 raft_consensus.cc:2726] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240501 20:01:28.724211 30754 raft_consensus.cc:482] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240501 20:01:28.744493 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 76205.6us
I20240501 20:01:28.672705 30756 raft_consensus.cc:482] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:28.671972 30768 raft_consensus.cc:482] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:28.728835 29729 raft_consensus.cc:2354] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d for earlier term 2. Current term is 3.
I20240501 20:01:28.745813 29733 raft_consensus.cc:3037] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:28.646785 30759 raft_consensus.cc:504] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:28.652397 30685 raft_consensus.cc:504] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:28.652343 30682 raft_consensus.cc:504] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:28.749724 29700 leader_election.cc:302] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:28.737795 30713 leader_election.cc:288] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:28.767087 29733 raft_consensus.cc:2454] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 3.
I20240501 20:01:28.730643 29807 raft_consensus.cc:2454] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:28.726704 30790 raft_consensus.cc:482] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:28.749650 30645 raft_consensus.cc:2781] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:01:28.744282 30761 raft_consensus.cc:482] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:28.655984 30771 raft_consensus.cc:504] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:28.802475 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31fff64b453a423eb0f567b3171564fb" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:28.802672 29732 raft_consensus.cc:1261] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:01:28.803172 29660 raft_consensus.cc:2454] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 2.
W20240501 20:01:28.772828 30773 log.cc:924] Time spent T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.241s user 0.001s sys 0.000s
I20240501 20:01:28.760280 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3f29f0eeea4b119984634202e5d9cd" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:28.803578 29734 raft_consensus.cc:2454] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
I20240501 20:01:28.812106 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5525f546426c47a38edf7f56ad04b7ab" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:28.813247 29625 leader_election.cc:302] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:28.813424 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e07220db7444ae97f294d70eda6a93" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:28.842581 29660 raft_consensus.cc:3037] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:28.843340 30420 raft_consensus.cc:1261] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240501 20:01:28.801724 30790 raft_consensus.cc:504] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:28.826632 29651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5525fc4b82a44b0991c3dd6098fde81" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
W20240501 20:01:28.826632 30793 log.cc:924] Time spent T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
W20240501 20:01:28.830694 30779 log.cc:924] Time spent T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 0.247s user 0.000s sys 0.001s
W20240501 20:01:28.831524 30785 log.cc:924] Time spent T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d: Append to log took a long time: real 0.196s user 0.000s sys 0.001s
W20240501 20:01:28.832304 30782 log.cc:924] Time spent T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d: Append to log took a long time: real 0.178s user 0.001s sys 0.000s
I20240501 20:01:28.757890 30756 raft_consensus.cc:504] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:28.773803 30706 raft_consensus.cc:482] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
W20240501 20:01:28.779271 30777 log.cc:924] Time spent T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.242s user 0.000s sys 0.001s
I20240501 20:01:28.834067 29731 raft_consensus.cc:2454] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 2.
I20240501 20:01:28.813838 29732 raft_consensus.cc:1261] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:01:28.836009 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05df43bdfa479b902b1da297fb5974" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:28.849880 29655 raft_consensus.cc:2354] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d for earlier term 2. Current term is 3.
I20240501 20:01:28.832921 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31fff64b453a423eb0f567b3171564fb" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:28.855713 29660 raft_consensus.cc:2454] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 888d99522e4844118f8f1cf26f268c8c in term 3.
I20240501 20:01:28.801844 30650 raft_consensus.cc:482] T 010ceabf3e5a4b1c9330b80dade765bb P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:28.750643 30385 leader_election.cc:302] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 888d99522e4844118f8f1cf26f268c8c, ee0d97247d394a8facfa774f15f3a67d; no voters: fdfeac4190e045f290e003b2298d988d
I20240501 20:01:28.752130 30754 raft_consensus.cc:504] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:28.768827 30768 raft_consensus.cc:504] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:28.751708 30788 raft_consensus.cc:504] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:28.889046 29652 raft_consensus.cc:3037] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:28.889144 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f5d1686214485d91a0baf4b870740e" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:28.889662 29733 raft_consensus.cc:2454] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 2.
I20240501 20:01:28.871155 30646 leader_election.cc:288] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:28.870702 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:28.814863 30761 raft_consensus.cc:504] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:29.004701 30713 raft_consensus.cc:2781] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:01:28.937088 30794 raft_consensus.cc:482] T 9046e38fc57f4364aa3d539e1fc01928 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
W20240501 20:01:28.936589 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 188467us
I20240501 20:01:28.938287 30760 raft_consensus.cc:2781] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:29.012944 30760 raft_consensus.cc:482] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:29.012632 30794 raft_consensus.cc:504] T 9046e38fc57f4364aa3d539e1fc01928 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:29.013074 29700 leader_election.cc:302] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:29.013396 30760 raft_consensus.cc:3037] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:29.013607 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05df43bdfa479b902b1da297fb5974" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:29.014379 29732 raft_consensus.cc:2454] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 888d99522e4844118f8f1cf26f268c8c in term 2.
I20240501 20:01:29.014735 30794 leader_election.cc:288] T 9046e38fc57f4364aa3d539e1fc01928 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:28.949262 30769 leader_election.cc:288] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:29.016269 29729 raft_consensus.cc:3037] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Advancing to term 3
W20240501 20:01:28.891171 30383 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 421037us
W20240501 20:01:29.016752 30385 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 272742us
I20240501 20:01:28.903182 30753 leader_election.cc:288] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:28.975942 29654 raft_consensus.cc:1226] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20240501 20:01:29.018911 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05df43bdfa479b902b1da297fb5974" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:29.019593 29804 raft_consensus.cc:2454] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 888d99522e4844118f8f1cf26f268c8c in term 1.
I20240501 20:01:29.019631 29806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41864 (request call id 748) took 1151 ms. Trace:
I20240501 20:01:28.895283 30650 raft_consensus.cc:504] T 010ceabf3e5a4b1c9330b80dade765bb P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:29.014609 29700 leader_election.cc:302] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:29.021299 30645 raft_consensus.cc:686] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [term 3 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:29.020386 29806 rpcz_store.cc:276] 0501 20:01:27.868186 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:27.868327 (+ 141us) service_pool.cc:225] Handling call
0501 20:01:27.868652 (+ 325us) raft_consensus.cc:1457] Updating replica for 1 ops
0501 20:01:27.868990 (+ 338us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:27.869020 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0501 20:01:27.869437 (+ 417us) log.cc:842] Serialized 26 byte log entry
0501 20:01:28.349455 (+480018us) raft_consensus.cc:1631] Marking committed up to 1
0501 20:01:28.349897 (+ 442us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:28.355231 (+ 5334us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0501 20:01:29.019333 (+664102us) raft_consensus.cc:1671] finished
0501 20:01:29.019368 (+ 35us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:29.019538 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":101,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":92,"thread_start_us":464156,"threads_started":1,"wal-append.queue_time_us":698485}
I20240501 20:01:28.887120 29670 tablet_copy_service.cc:342] P 4c8da1d580614449a636b855bf11b95f: Request end of tablet copy session 888d99522e4844118f8f1cf26f268c8c-be383be783234855878cb8fd442c7b05 received from {username='slave'} at 127.0.0.1:37936
I20240501 20:01:29.022258 30645 consensus_queue.cc:237] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:29.022972 29670 tablet_copy_service.cc:433] P 4c8da1d580614449a636b855bf11b95f: ending tablet copy session 888d99522e4844118f8f1cf26f268c8c-be383be783234855878cb8fd442c7b05 on tablet be383be783234855878cb8fd442c7b05 with peer 888d99522e4844118f8f1cf26f268c8c
I20240501 20:01:29.023468 29804 raft_consensus.cc:1261] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:01:28.896106 29651 raft_consensus.cc:2354] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d for earlier term 2. Current term is 3.
I20240501 20:01:28.891855 30706 raft_consensus.cc:504] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:29.027369 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcf30cf66714efdac997862dafbe4a2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:28.875491 30762 raft_consensus.cc:482] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:29.029389 29732 raft_consensus.cc:1226] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 2. Current term is 3. Ops: []
I20240501 20:01:29.015758 30794 raft_consensus.cc:2781] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:28.899749 29660 raft_consensus.cc:3037] T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 3
I20240501 20:01:29.040410 29652 raft_consensus.cc:2454] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 3.
I20240501 20:01:29.045578 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07bdb8b09b8c4ef4b8e49daa9b8c01d6" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:28.859719 30654 raft_consensus.cc:482] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:28.903532 30740 leader_election.cc:288] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:29.022308 30650 leader_election.cc:288] T 010ceabf3e5a4b1c9330b80dade765bb P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:28.951958 30801 raft_consensus.cc:482] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:29.021006 30385 leader_election.cc:302] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 888d99522e4844118f8f1cf26f268c8c, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:29.052440 30703 leader_election.cc:288] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:29.095533 29625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 282440us
I20240501 20:01:29.709152 29732 raft_consensus.cc:1261] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:01:29.088351 30751 raft_consensus.cc:482] T 1ca3948261374cef8b1b206a43c00173 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:29.710369 30751 raft_consensus.cc:504] T 1ca3948261374cef8b1b206a43c00173 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:29.712555 30751 leader_election.cc:288] T 1ca3948261374cef8b1b206a43c00173 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
W20240501 20:01:29.699293 30760 consensus_meta.cc:287] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c: Time spent flushing consensus metadata: real 0.685s user 0.006s sys 0.005s
I20240501 20:01:29.712618 30383 leader_election.cc:302] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, 888d99522e4844118f8f1cf26f268c8c; no voters: fdfeac4190e045f290e003b2298d988d
W20240501 20:01:29.707839 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 660911us
I20240501 20:01:29.681984 30706 leader_election.cc:288] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:29.714888 30817 raft_consensus.cc:482] T 84fa28fb55f44b61ba1a88ed775a78ba P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:29.715387 30817 raft_consensus.cc:504] T 84fa28fb55f44b61ba1a88ed775a78ba P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:29.081138 30796 raft_consensus.cc:482] T 3c5345f6eb62449ba69b4a9c2e349b31 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:29.716655 30796 raft_consensus.cc:504] T 3c5345f6eb62449ba69b4a9c2e349b31 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:29.717439 30817 leader_election.cc:288] T 84fa28fb55f44b61ba1a88ed775a78ba P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
W20240501 20:01:29.656123 30780 log.cc:924] Time spent T 456035a1cb3f4f7da52d0fc1dd1c656c P ee0d97247d394a8facfa774f15f3a67d: Append to log took a long time: real 0.353s user 0.000s sys 0.001s
I20240501 20:01:29.084623 29660 raft_consensus.cc:2454] T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 3.
I20240501 20:01:29.052743 30753 raft_consensus.cc:2781] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:29.728688 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0374bb790ed345ed90452595f71383a2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:29.712747 29625 leader_election.cc:302] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:29.746227 29731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 662) took 1145 ms. Trace:
I20240501 20:01:29.718959 29809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41864 (request call id 744) took 1987 ms. Trace:
I20240501 20:01:29.746266 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49da4246f5f6438abe0e7e8c9a523d80" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:29.773098 29655 raft_consensus.cc:2379] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 2: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:29.764760 29731 rpcz_store.cc:276] 0501 20:01:28.601218 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:28.624859 (+ 23641us) service_pool.cc:225] Handling call
0501 20:01:29.745990 (+1121131us) spinlock_profiling.cc:243] Waited 166 ms on lock 0x7b5400154854. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787f4fe 00007fe45c7bd9d7 00007fe455ea1c14 00007fe455ea1ba2 00007fe455ea1ad1 00007fe455ea1a4c 00007fe455ea0d02 00007fe455ca2bcc 00007fe455ca2006 00007fe455ca19af
0501 20:01:29.746129 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":380669312}
I20240501 20:01:29.776584 30800 raft_consensus.cc:482] T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:29.777240 30800 raft_consensus.cc:504] T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:29.779531 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e49dd5180f04313ab33942fb3ddc147" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:29.773109 29809 rpcz_store.cc:276] 0501 20:01:27.732641 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:27.735774 (+ 3133us) service_pool.cc:225] Handling call
0501 20:01:27.736194 (+ 420us) raft_consensus.cc:1457] Updating replica for 1 ops
0501 20:01:27.748537 (+ 12343us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:27.748566 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0501 20:01:27.748943 (+ 377us) log.cc:842] Serialized 26 byte log entry
0501 20:01:28.418088 (+669145us) raft_consensus.cc:1631] Marking committed up to 1
0501 20:01:28.418135 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:28.418224 (+ 89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0501 20:01:29.718741 (+1300517us) raft_consensus.cc:1671] finished
0501 20:01:29.718764 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:29.718894 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11096,"raft.queue_time_us":11192,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":94,"thread_start_us":668779,"threads_started":1,"wal-append.queue_time_us":895861}
I20240501 20:01:29.779484 29657 raft_consensus.cc:2379] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 3: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
W20240501 20:01:29.081390 30774 log.cc:924] Time spent T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.303s user 0.001s sys 0.001s
I20240501 20:01:29.033142 30794 raft_consensus.cc:482] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
W20240501 20:01:29.081547 30791 log.cc:924] Time spent T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
I20240501 20:01:29.800175 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece8d4ed84c34b748f7b6272b31c90ac" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:29.292152 30685 leader_election.cc:288] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:29.325184 30804 log.cc:924] Time spent T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.164s user 0.000s sys 0.000s
W20240501 20:01:29.464437 30776 log.cc:924] Time spent T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d: Append to log took a long time: real 0.897s user 0.000s sys 0.005s
W20240501 20:01:29.801138 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.196:37227, user_credentials={real_user=slave}} blocked reactor thread for 721901us
W20240501 20:01:29.757072 30418 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.811s user 0.004s sys 0.016s
W20240501 20:01:29.081220 30772 log.cc:924] Time spent T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d: Append to log took a long time: real 0.266s user 0.000s sys 0.001s
W20240501 20:01:29.739953 29729 consensus_meta.cc:287] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d: Time spent flushing consensus metadata: real 0.723s user 0.008s sys 0.005s
W20240501 20:01:29.083448 30792 log.cc:924] Time spent T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 0.218s user 0.000s sys 0.000s
I20240501 20:01:29.449337 30713 raft_consensus.cc:686] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:29.074308 30654 raft_consensus.cc:504] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:29.078583 29805 raft_consensus.cc:2454] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 3.
I20240501 20:01:29.806084 29651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6797d82dda2948d8a7445fcd33e44752" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:29.806136 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcf30cf66714efdac997862dafbe4a2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:29.726433 30809 raft_consensus.cc:482] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:29.807740 29726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 641) took 2548 ms. Trace:
I20240501 20:01:29.728976 30706 raft_consensus.cc:482] T 2d86f8848a184a87947fd927c44e5a35 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:29.092339 30649 leader_election.cc:288] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:29.470211 30790 leader_election.cc:288] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:29.745553 30821 raft_consensus.cc:2781] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:01:29.916105 30821 raft_consensus.cc:686] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [term 3 LEADER]: Becoming Leader. State: Replica: 888d99522e4844118f8f1cf26f268c8c, State: Running, Role: LEADER
I20240501 20:01:29.916247 30706 raft_consensus.cc:504] T 2d86f8848a184a87947fd927c44e5a35 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:29.917126 30821 consensus_queue.cc:237] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:29.918040 30654 leader_election.cc:288] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:29.918411 30706 leader_election.cc:288] T 2d86f8848a184a87947fd927c44e5a35 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:29.918718 29658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59142 (request call id 749) took 1115 ms. Trace:
I20240501 20:01:29.918849 30809 raft_consensus.cc:504] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:29.854451 30383 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 117485us
I20240501 20:01:29.920236 30788 leader_election.cc:288] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:29.919009 29726 rpcz_store.cc:276] 0501 20:01:27.259688 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:27.259876 (+ 188us) service_pool.cc:225] Handling call
0501 20:01:27.358808 (+ 98932us) spinlock_profiling.cc:243] Waited 11.5 ms on lock 0x7b58001b3098. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe454797b51 00007fe4549193b1 00007fe454919262 00007fe45d80b889 00007fe45c7483c0 00007fe45c80e3bb 00007fe45c80e441 00007fe45c7d2060 00007fe45c7bd5a9 00007fe45c7bd052 00007fe455e9f694
0501 20:01:27.359007 (+ 199us) raft_consensus.cc:1457] Updating replica for 1 ops
0501 20:01:27.725340 (+366333us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:27.725371 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0501 20:01:27.726883 (+ 1512us) log.cc:842] Serialized 26 byte log entry
0501 20:01:28.223245 (+496362us) raft_consensus.cc:1631] Marking committed up to 2
0501 20:01:28.259781 (+ 36536us) log.cc:842] Serialized 14 byte log entry
0501 20:01:28.260141 (+ 360us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:28.260240 (+ 99us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0501 20:01:28.372116 (+111876us) spinlock_profiling.cc:243] Waited 5.41 ms on lock 0x7b1800185320. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe45d7f2661 00007fe45d7ff3e1 00007fe4549511b8 00007fe454954691 00007fe454954649 00007fe4549545d9 00007fe4549545a1 00007fe45495389d 00007fe45dbc5da4 00007fe45dbc5bd9 00007fe454933936
0501 20:01:29.807374 (+1435258us) raft_consensus.cc:1671] finished
0501 20:01:29.807410 (+ 36us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:29.807611 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":28816,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":31736,"raft.queue_time_us":46030,"raft.run_cpu_time_us":297,"raft.run_wall_time_us":23781,"spinlock_wait_cycles":39005952,"thread_start_us":855002,"threads_started":2,"wal-append.queue_time_us":690243}
I20240501 20:01:29.658636 30754 leader_election.cc:288] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:29.331779 30748 leader_election.cc:288] T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:29.190752 30759 leader_election.cc:288] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:29.755349 30753 raft_consensus.cc:482] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:29.718708 30796 leader_election.cc:288] T 3c5345f6eb62449ba69b4a9c2e349b31 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:29.076272 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfac893b4b4f457694a4eb184b6abb53" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:29.933549 29651 raft_consensus.cc:2454] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 2.
I20240501 20:01:29.943959 30759 raft_consensus.cc:482] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:29.944466 29651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:29.944532 30759 raft_consensus.cc:504] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:29.945274 29651 raft_consensus.cc:3037] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 3
I20240501 20:01:29.946799 30759 leader_election.cc:288] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:29.800174 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a35893b91247a99912eba787d787cf" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:29.948194 29660 raft_consensus.cc:2379] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 2: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:29.948972 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649b71f9f267437ca51c758dd9ca6987" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:29.949540 29660 raft_consensus.cc:3037] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:29.082902 30795 raft_consensus.cc:482] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:29.952415 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649b71f9f267437ca51c758dd9ca6987" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:29.952337 30795 raft_consensus.cc:504] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:29.952606 30713 consensus_queue.cc:237] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:29.953718 29805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41858 (request call id 720) took 1357 ms. Trace:
I20240501 20:01:29.042820 30762 raft_consensus.cc:504] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:29.903589 29772 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.718s user 0.015s sys 0.050s
W20240501 20:01:29.854543 29625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 141852us
W20240501 20:01:29.956245 29772 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.770s user 0.016s sys 0.050s
I20240501 20:01:29.726971 30808 consensus_queue.cc:1035] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.168s
I20240501 20:01:29.933014 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5525f546426c47a38edf7f56ad04b7ab" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:29.960749 29655 raft_consensus.cc:2379] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 3: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:29.911645 30650 raft_consensus.cc:482] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:29.965525 30650 raft_consensus.cc:504] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:29.966367 30762 leader_election.cc:288] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:29.948210 30809 leader_election.cc:288] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:29.737504 30815 raft_consensus.cc:482] T d1d2cac25f354a97b1bfb53d57ae7c08 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:29.951788 30682 leader_election.cc:288] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:29.774533 30761 leader_election.cc:288] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:29.185433 30613 raft_consensus.cc:482] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:29.728679 30820 raft_consensus.cc:482] T ec073fe645614fe28e7837de3c22510b P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:29.877454 30646 raft_consensus.cc:2781] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election won for term 3
W20240501 20:01:29.881942 30385 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 860953us
I20240501 20:01:29.919102 29658 rpcz_store.cc:276] 0501 20:01:28.803504 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:28.825706 (+ 22202us) service_pool.cc:225] Handling call
0501 20:01:28.847263 (+ 21557us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:29.168864 (+321601us) raft_consensus.cc:1520] Early marking committed up to index 5
0501 20:01:29.462156 (+293292us) log.cc:842] Serialized 14 byte log entry
0501 20:01:29.799515 (+337359us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:29.804262 (+ 4747us) raft_consensus.cc:1631] Marking committed up to 5
0501 20:01:29.806639 (+ 2377us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:29.918462 (+111823us) spinlock_profiling.cc:243] Waited 241 ms on lock 0x7b54003958d4. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787c1a1 00007fe45787abee 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc 00007fe455e9e782 00007fe455ca2bcc 00007fe455ca2006
0501 20:01:29.918508 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:29.918639 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":135231,"prepare.queue_time_us":549970,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":50,"raft.queue_time_us":66726,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":2546,"spinlock_wait_cycles":554528512,"thread_start_us":402069,"threads_started":2,"wal-append.queue_time_us":270756}
I20240501 20:01:29.953153 29728 raft_consensus.cc:3037] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:29.713340 30760 raft_consensus.cc:504] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:29.081732 30798 raft_consensus.cc:2781] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election won for term 3
W20240501 20:01:29.854708 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 93349us
I20240501 20:01:29.798898 30740 raft_consensus.cc:2781] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:29.815717 30794 raft_consensus.cc:3037] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:01:29.925895 30418 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.980s user 0.009s sys 0.019s
I20240501 20:01:29.959812 29625 leader_election.cc:302] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:29.930390 29554 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:43384 (request call id 143) took 1641 ms. Trace:
I20240501 20:01:29.914932 30769 raft_consensus.cc:482] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:29.911283 30805 raft_consensus.cc:482] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:29.915155 30807 consensus_queue.cc:1035] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.157s
I20240501 20:01:29.964879 29655 raft_consensus.cc:1261] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:01:29.923880 29729 raft_consensus.cc:2454] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 3.
I20240501 20:01:29.965852 29802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41864 (request call id 739) took 2381 ms. Trace:
I20240501 20:01:29.800199 30817 raft_consensus.cc:482] T a8787bc4d6b44649b9138d906730110a P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:29.088564 30756 leader_election.cc:288] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:29.089756 30771 leader_election.cc:288] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:29.938253 30753 raft_consensus.cc:3037] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:29.954119 29805 rpcz_store.cc:276] 0501 20:01:28.571326 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:28.573304 (+ 1978us) service_pool.cc:225] Handling call
0501 20:01:29.928595 (+1355291us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:29.910949 30806 consensus_queue.cc:1046] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.081s
I20240501 20:01:29.780488 30801 raft_consensus.cc:504] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:30.199091 29772 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.27.217.193:33959 timed out after 2.418s (SENT) [suppressed 13 similar messages]
I20240501 20:01:29.969831 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211e27a3b0084b89a6dd7550849b5a65" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:29.999691 29660 raft_consensus.cc:2454] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 3.
I20240501 20:01:29.984721 29801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac2171683884f6f97cdfeb0a656cb49" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:30.238215 29660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59136 (request call id 639) took 1193 ms. Trace:
I20240501 20:01:29.971774 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af08384057a4d85977481472f460168" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:30.204916 30809 raft_consensus.cc:482] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:30.216404 30761 raft_consensus.cc:482] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:30.205767 29735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 667) took 1543 ms. Trace:
I20240501 20:01:30.205456 30769 raft_consensus.cc:504] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:30.206050 30805 raft_consensus.cc:504] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:29.979008 30645 consensus_queue.cc:1046] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.228s
I20240501 20:01:30.207363 30740 raft_consensus.cc:482] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:30.207103 30817 raft_consensus.cc:504] T a8787bc4d6b44649b9138d906730110a P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:30.064191 30703 raft_consensus.cc:482] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240501 20:01:30.208140 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.196:37227, user_credentials={real_user=slave}} blocked reactor thread for 281716us
I20240501 20:01:30.240775 29735 rpcz_store.cc:276] 0501 20:01:28.662833 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:28.662950 (+ 117us) service_pool.cc:225] Handling call
0501 20:01:28.663228 (+ 278us) raft_consensus.cc:1457] Updating replica for 1 ops
0501 20:01:28.663545 (+ 317us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:28.663571 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0501 20:01:28.664029 (+ 458us) log.cc:842] Serialized 26 byte log entry
0501 20:01:28.860078 (+196049us) raft_consensus.cc:1631] Marking committed up to 1
0501 20:01:28.860145 (+ 67us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:28.860241 (+ 96us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0501 20:01:30.205525 (+1345284us) raft_consensus.cc:1671] finished
0501 20:01:30.205547 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:30.205690 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":692,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":12520,"thread_start_us":129933,"threads_started":1,"wal-append.queue_time_us":306875}
I20240501 20:01:30.208371 29655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59136 (request call id 637) took 1164 ms. Trace:
I20240501 20:01:30.208791 29658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59136 (request call id 638) took 1164 ms. Trace:
I20240501 20:01:30.236338 30814 raft_consensus.cc:482] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 888d99522e4844118f8f1cf26f268c8c)
I20240501 20:01:30.209674 29657 raft_consensus.cc:2454] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 2.
I20240501 20:01:30.215656 30613 raft_consensus.cc:504] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:30.142699 30754 raft_consensus.cc:482] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:30.258836 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668b4a4d782c4278932dd8413dc3ab3e" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:30.257640 29730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 677) took 1394 ms. Trace:
I20240501 20:01:30.258358 29655 rpcz_store.cc:276] 0501 20:01:29.043752 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:29.964260 (+920508us) service_pool.cc:225] Handling call
0501 20:01:29.964582 (+ 322us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:30.208125 (+243543us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:30.208323 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:30.211454 30807 raft_consensus.cc:482] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
W20240501 20:01:30.075493 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 320181us
I20240501 20:01:30.096849 30796 raft_consensus.cc:482] T f6d88139f89c47ff84cad674147aae0f P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:30.327252 30796 raft_consensus.cc:504] T f6d88139f89c47ff84cad674147aae0f P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:30.328122 29655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59136 (request call id 640) took 1283 ms. Trace:
I20240501 20:01:30.327143 29730 rpcz_store.cc:276] 0501 20:01:28.862912 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:28.863076 (+ 164us) service_pool.cc:225] Handling call
0501 20:01:28.863371 (+ 295us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:28.894116 (+ 30745us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:28.894152 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:28.894193 (+ 41us) raft_consensus.cc:1631] Marking committed up to 1
0501 20:01:28.894222 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:30.252348 (+1358126us) spinlock_profiling.cc:243] Waited 30.6 ms on lock 0x7b5400290254. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787c1a1 00007fe45787abee 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc 00007fe455e9e782 00007fe455ca2bcc 00007fe455ca2006
0501 20:01:30.252422 (+ 74us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:30.257552 (+ 5130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":70280064}
I20240501 20:01:30.253650 29654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59142 (request call id 746) took 1451 ms. Trace:
I20240501 20:01:30.329900 30796 leader_election.cc:288] T f6d88139f89c47ff84cad674147aae0f P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:30.108985 30685 raft_consensus.cc:482] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:30.329973 29654 rpcz_store.cc:276] 0501 20:01:28.802696 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:28.810283 (+ 7587us) service_pool.cc:225] Handling call
0501 20:01:28.894025 (+ 83742us) raft_consensus.cc:1457] Updating replica for 1 ops
0501 20:01:29.018456 (+124431us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:30.253386 (+1234930us) spinlock_profiling.cc:243] Waited 34.8 ms on lock 0x7b54001c2350. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787ac59 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc 00007fe455e9e782 00007fe455ca2bcc 00007fe455ca2006 00007fe455ca19af
0501 20:01:30.253560 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":80009856}
I20240501 20:01:29.954597 30795 leader_election.cc:288] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
W20240501 20:01:30.216648 29625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 256903us
I20240501 20:01:30.330978 30685 raft_consensus.cc:504] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:30.331764 29654 raft_consensus.cc:1261] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:01:30.221215 29624 leader_election.cc:302] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
W20240501 20:01:30.335376 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 114255us
I20240501 20:01:30.209128 30418 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40670 (request call id 32) took 1946 ms. Trace:
I20240501 20:01:30.142182 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05df43bdfa479b902b1da297fb5974" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:30.238570 29660 rpcz_store.cc:276] 0501 20:01:29.044421 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:29.948723 (+904302us) service_pool.cc:225] Handling call
0501 20:01:30.238159 (+289436us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:30.340451 29660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59136 (request call id 643) took 1289 ms. Trace:
I20240501 20:01:30.340902 29660 rpcz_store.cc:276] 0501 20:01:29.051415 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:30.339429 (+1288014us) service_pool.cc:225] Handling call
0501 20:01:30.339808 (+ 379us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:30.340015 (+ 207us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:30.340075 (+ 60us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:30.340133 (+ 58us) raft_consensus.cc:1631] Marking committed up to 1
0501 20:01:30.340176 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:30.340247 (+ 71us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:30.340383 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:30.342564 29660 raft_consensus.cc:3037] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:30.209642 30820 raft_consensus.cc:504] T ec073fe645614fe28e7837de3c22510b P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:30.203714 30650 leader_election.cc:288] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:30.063521 30682 raft_consensus.cc:482] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:30.220031 30756 raft_consensus.cc:482] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:30.220782 30787 raft_consensus.cc:2781] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:30.208858 29729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 672) took 1398 ms. Trace:
I20240501 20:01:29.980532 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6797d82dda2948d8a7445fcd33e44752" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:30.355520 29808 raft_consensus.cc:2454] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 2.
I20240501 20:01:30.356526 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9472c53f79ec423f82c396bd980674e2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:30.357393 29700 leader_election.cc:302] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:30.244014 30829 raft_consensus.cc:482] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:30.303350 29727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 682) took 1247 ms. Trace:
I20240501 20:01:30.328508 29655 rpcz_store.cc:276] 0501 20:01:29.044717 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:30.327329 (+1282612us) service_pool.cc:225] Handling call
0501 20:01:30.327643 (+ 314us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:30.327798 (+ 155us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:30.327822 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:30.327861 (+ 39us) raft_consensus.cc:1631] Marking committed up to 1
0501 20:01:30.327891 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:30.327940 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:30.328072 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:30.363425 29655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59136 (request call id 645) took 1307 ms. Trace:
I20240501 20:01:30.363744 29655 rpcz_store.cc:276] 0501 20:01:29.055561 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:30.362663 (+1307102us) service_pool.cc:225] Handling call
0501 20:01:30.362975 (+ 312us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:30.363137 (+ 162us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:30.363161 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:30.363202 (+ 41us) raft_consensus.cc:1631] Marking committed up to 1
0501 20:01:30.363230 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:30.363280 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:30.363375 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:30.365267 29655 raft_consensus.cc:3037] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:30.213944 29554 rpcz_store.cc:276] 0501 20:01:28.281045 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:28.281227 (+ 182us) service_pool.cc:225] Handling call
0501 20:01:28.804341 (+523114us) spinlock_profiling.cc:243] Waited 256 ms on lock 0x7b54000037d4. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe457800d30 00007fe45788417a 00007fe45d7d5b1c 00007fe45d8cfa8e 00007fe45bbe3a44 00007fe45bbe39d2 00007fe45bbe3901 00007fe45bbe387c 00007fe45bbe2b32 00007fe455ca2bcc 00007fe455ca2006
0501 20:01:29.922080 (+1117739us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":3,"spinlock_wait_cycles":587860736}
I20240501 20:01:30.261021 29658 rpcz_store.cc:276] 0501 20:01:29.044096 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:30.207930 (+1163834us) service_pool.cc:225] Handling call
0501 20:01:30.208311 (+ 381us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:30.208475 (+ 164us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:30.208501 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:30.208546 (+ 45us) raft_consensus.cc:1631] Marking committed up to 1
0501 20:01:30.208577 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:30.208632 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:30.208739 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:30.032681 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8787bc4d6b44649b9138d906730110a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:30.332966 30685 leader_election.cc:288] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:30.332851 30794 raft_consensus.cc:504] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:30.338352 29624 leader_election.cc:302] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:30.345259 30410 tablet_service.cc:1514] Processing DeleteTablet for tablet be383be783234855878cb8fd442c7b05 with delete_type TABLET_DATA_TOMBSTONED (TS 888d99522e4844118f8f1cf26f268c8c not found in new config with opid_index -1) from {username='slave'} at 127.0.0.1:45668
I20240501 20:01:30.304828 30646 raft_consensus.cc:482] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:30.314693 30645 raft_consensus.cc:482] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:30.345077 30418 rpcz_store.cc:276] 0501 20:01:28.263028 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:28.264174 (+ 1146us) service_pool.cc:225] Handling call
0501 20:01:28.264477 (+ 303us) raft_consensus.cc:1457] Updating replica for 1 ops
0501 20:01:28.264799 (+ 322us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:28.264828 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0501 20:01:28.834210 (+569382us) spinlock_profiling.cc:243] Waited 1.25 ms on lock 0x7b2c000d6e58. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe45625df11 00007fe45626d542 00007fe4578b6fc1 00007fe45787b7da 00007fe45787abee 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc 00007fe455e9e782
0501 20:01:28.894832 (+ 60622us) log.cc:842] Serialized 26 byte log entry
0501 20:01:30.208657 (+1313825us) raft_consensus.cc:1631] Marking committed up to 1
0501 20:01:30.208700 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:30.208771 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0501 20:01:30.208822 (+ 51us) raft_consensus.cc:1671] finished
0501 20:01:30.208833 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:30.209043 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":553,"raft.run_cpu_time_us":44,"raft.run_wall_time_us":44,"spinlock_wait_cycles":2881280,"thread_start_us":1270943,"threads_started":1,"wal-append.queue_time_us":830006}
I20240501 20:01:30.244534 30762 raft_consensus.cc:3032] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:01:30.013053 30815 raft_consensus.cc:504] T d1d2cac25f354a97b1bfb53d57ae7c08 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:30.242835 30809 raft_consensus.cc:504] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } }
I20240501 20:01:30.209573 29802 rpcz_store.cc:276] 0501 20:01:27.584153 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:27.585716 (+ 1563us) service_pool.cc:225] Handling call
0501 20:01:27.689403 (+103687us) spinlock_profiling.cc:243] Waited 11.2 ms on lock 0x7b58000d2c98. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe454797b51 00007fe4549196dc 00007fe45491929b 00007fe45d80b889 00007fe45c7483c0 00007fe45c80e3bb 00007fe45c80e441 00007fe45c7d2060 00007fe45c7bd5a9 00007fe45c7bd052 00007fe455e9f694
0501 20:01:27.689580 (+ 177us) raft_consensus.cc:1457] Updating replica for 1 ops
0501 20:01:27.725826 (+ 36246us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:27.725860 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0501 20:01:27.726323 (+ 463us) log.cc:842] Serialized 26 byte log entry
0501 20:01:27.776163 (+ 49840us) spinlock_profiling.cc:243] Waited 3.21 ms on lock 0x7b4800061158. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe45c723431 00007fe45c71a527 00007fe45c71a3ba 00007fe45c80e57f 00007fe45c80e247 00007fe45c82274c 00007fe45c8226f8 00007fe45c822678 00007fe45c822633 00007fe45c8218dc 00007fe45b7a49b6
0501 20:01:28.381628 (+605465us) raft_consensus.cc:1631] Marking committed up to 1
0501 20:01:28.384925 (+ 3297us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:28.401469 (+ 16544us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0501 20:01:29.961314 (+1559845us) raft_consensus.cc:1671] finished
0501 20:01:29.965644 (+ 4330us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:29.965772 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34811,"raft.queue_time_us":37730,"raft.run_cpu_time_us":2595,"raft.run_wall_time_us":48618,"spinlock_wait_cycles":33070336,"thread_start_us":637194,"threads_started":1,"wal-append.queue_time_us":743536}
I20240501 20:01:30.063822 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1a08b455b94ae089396a15c837ddc2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:30.308593 30751 raft_consensus.cc:482] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:30.244175 30761 raft_consensus.cc:504] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:30.346645 30801 leader_election.cc:288] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:30.249879 30740 raft_consensus.cc:3037] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:30.317373 30798 raft_consensus.cc:686] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [term 3 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
W20240501 20:01:30.345114 30783 log.cc:924] Time spent T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d: Append to log took a long time: real 1.673s user 0.005s sys 0.007s
I20240501 20:01:30.305264 30771 raft_consensus.cc:482] T c651044a305c4168805825a53d910899 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:30.246058 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "696121417ea9403990456cfc6fc4f416" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:30.305688 29651 raft_consensus.cc:2454] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 3.
I20240501 20:01:30.369052 29803 raft_consensus.cc:2454] T a8787bc4d6b44649b9138d906730110a P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
I20240501 20:01:30.372123 30812 raft_consensus.cc:2781] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:30.346639 30836 raft_consensus.cc:3032] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:01:30.251211 30827 raft_consensus.cc:482] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:30.380863 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48fd0a8e289401aa7417c42ff2c81b2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:30.452898 29700 leader_election.cc:302] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:30.382105 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca3948261374cef8b1b206a43c00173" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:30.382874 29659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59142 (request call id 748) took 1579 ms. Trace:
W20240501 20:01:30.457116 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 96540us
I20240501 20:01:30.431223 30760 leader_election.cc:288] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 2 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:30.456956 29659 rpcz_store.cc:276] 0501 20:01:28.803238 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:28.808571 (+ 5333us) service_pool.cc:225] Handling call
0501 20:01:29.445780 (+637209us) spinlock_profiling.cc:243] Waited 3.92 ms on lock 0x7b5800001598. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe454797b51 00007fe4549193b1 00007fe454919262 00007fe45d80b889 00007fe45c7483c0 00007fe45c80e3bb 00007fe45c80e441 00007fe45c7d2060 00007fe45c7bd5a9 00007fe45c7bd052 00007fe455e9f694
0501 20:01:29.446047 (+ 267us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:29.446297 (+ 250us) raft_consensus.cc:1520] Early marking committed up to index 6
0501 20:01:29.744097 (+297800us) log.cc:842] Serialized 14 byte log entry
0501 20:01:30.216594 (+472497us) spinlock_profiling.cc:243] Waited 19.2 ms on lock 0x7b180038b1e0. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe45d7f2661 00007fe45d7ff3e1 00007fe454932eb5 00007fe45d76c939 00007fe45495042b 00007fe45494e82f 00007fe45494dcc5 00007fe457873620 00007fe4578857c6 00007fe457896327 00007fe4578962a8
0501 20:01:30.382574 (+165980us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:30.382626 (+ 52us) raft_consensus.cc:1631] Marking committed up to 6
0501 20:01:30.382666 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:30.382724 (+ 58us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:30.382825 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":61071,"prepare.queue_time_us":292574,"prepare.run_cpu_time_us":24,"prepare.run_wall_time_us":25,"raft.queue_time_us":84550,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":91,"spinlock_wait_cycles":53257216,"thread_start_us":469161,"threads_started":3,"wal-append.queue_time_us":153292}
W20240501 20:01:30.383988 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 45743.4us
I20240501 20:01:30.354326 30787 raft_consensus.cc:482] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:30.314069 30834 raft_consensus.cc:482] T 1ca3948261374cef8b1b206a43c00173 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:30.355165 29804 raft_consensus.cc:2454] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
I20240501 20:01:30.362195 30796 raft_consensus.cc:482] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:30.475201 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b337b8e3e38d4ad6b1076fbd0d0a1609" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:30.390830 29660 raft_consensus.cc:1261] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:01:30.384423 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d88139f89c47ff84cad674147aae0f" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:30.416451 29733 raft_consensus.cc:2454] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 2.
I20240501 20:01:30.309186 30814 raft_consensus.cc:504] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:30.344278 30754 raft_consensus.cc:504] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:30.311949 30703 raft_consensus.cc:504] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:30.423851 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5345f6eb62449ba69b4a9c2e349b31" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:30.495306 29731 raft_consensus.cc:2454] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 2.
I20240501 20:01:30.426932 29698 leader_election.cc:398] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Vote denied by peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959) with higher term. Message: Invalid argument: T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d for earlier term 2. Current term is 3.
W20240501 20:01:30.496582 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 79501.1us
I20240501 20:01:30.437441 30751 raft_consensus.cc:504] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:30.428921 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84fa28fb55f44b61ba1a88ed775a78ba" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:30.429015 29625 leader_election.cc:302] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
W20240501 20:01:30.506826 29625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 77944us
I20240501 20:01:30.455816 29805 raft_consensus.cc:2454] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 2.
I20240501 20:01:30.384883 30646 raft_consensus.cc:3037] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:30.387529 30790 raft_consensus.cc:2781] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:01:30.360972 30756 raft_consensus.cc:504] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:30.354521 29729 rpcz_store.cc:276] 0501 20:01:28.810575 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:28.815312 (+ 4737us) service_pool.cc:225] Handling call
0501 20:01:30.208797 (+1393485us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:30.522146 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a35893b91247a99912eba787d787cf" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:30.523115 29731 raft_consensus.cc:2454] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 2.
I20240501 20:01:30.355283 30807 raft_consensus.cc:504] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:30.414963 30762 raft_consensus.cc:729] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:30.372862 30829 raft_consensus.cc:504] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:30.387768 30805 leader_election.cc:288] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:30.450734 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1a08b455b94ae089396a15c837ddc2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:30.456581 29730 raft_consensus.cc:2454] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 3.
I20240501 20:01:30.415195 30645 raft_consensus.cc:504] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:30.533854 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e9f75387564275a60d38e9a910319c" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:30.534648 29730 raft_consensus.cc:2454] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:30.535653 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5525f546426c47a38edf7f56ad04b7ab" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:30.536952 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48fd0a8e289401aa7417c42ff2c81b2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:30.537621 29730 raft_consensus.cc:2454] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 3.
I20240501 20:01:30.455960 29658 raft_consensus.cc:3037] T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:30.480803 29660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59136 (request call id 644) took 1429 ms. Trace:
I20240501 20:01:30.538538 29660 rpcz_store.cc:276] 0501 20:01:29.051674 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:30.342079 (+1290405us) service_pool.cc:225] Handling call
0501 20:01:30.342437 (+ 358us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:30.480445 (+138008us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:30.480729 (+ 284us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:30.512562 30748 raft_consensus.cc:2781] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:01:30.497802 29698 leader_election.cc:398] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Vote denied by peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959) with higher term. Message: Invalid argument: T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d for earlier term 2. Current term is 3.
I20240501 20:01:30.374907 29654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59136 (request call id 642) took 1323 ms. Trace:
I20240501 20:01:30.498540 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6797d82dda2948d8a7445fcd33e44752" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:30.470057 30812 raft_consensus.cc:482] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:30.471988 30787 raft_consensus.cc:3037] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:30.459031 29651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59136 (request call id 660) took 1141 ms. Trace:
I20240501 20:01:30.469471 30828 raft_consensus.cc:482] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:30.540397 29732 raft_consensus.cc:2454] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
I20240501 20:01:30.506192 29726 raft_consensus.cc:2454] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
I20240501 20:01:30.473671 29802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfac893b4b4f457694a4eb184b6abb53" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:30.472699 30836 raft_consensus.cc:729] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:30.449604 30771 raft_consensus.cc:504] T c651044a305c4168805825a53d910899 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:30.556865 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38cbeb0dc0c14e3d9182101eea885625" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:30.558118 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668b4a4d782c4278932dd8413dc3ab3e" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:30.559181 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48fd0a8e289401aa7417c42ff2c81b2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:30.560743 29802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af08384057a4d85977481472f460168" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:30.560914 30768 leader_election.cc:288] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:30.561815 30768 raft_consensus.cc:482] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:30.561933 29802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5525f546426c47a38edf7f56ad04b7ab" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:30.562471 30768 raft_consensus.cc:504] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:30.563114 29802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6797d82dda2948d8a7445fcd33e44752" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:30.563745 29802 raft_consensus.cc:2454] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 2.
I20240501 20:01:30.482105 30769 leader_election.cc:288] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:30.529448 30832 raft_consensus.cc:2781] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:30.539654 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5345f6eb62449ba69b4a9c2e349b31" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:30.487008 29700 leader_election.cc:302] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
W20240501 20:01:30.570101 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 83189.4us
W20240501 20:01:30.541975 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 83695.3us
I20240501 20:01:30.542770 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af08384057a4d85977481472f460168" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:30.513453 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48630ae6eb394ac1b54955dd01cf0ed2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:30.469740 30827 raft_consensus.cc:504] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:30.516333 30810 raft_consensus.cc:482] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:30.483554 30796 raft_consensus.cc:504] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:30.514438 30650 raft_consensus.cc:482] T add9d7579351483fb268f89a44e95d8a P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 888d99522e4844118f8f1cf26f268c8c)
I20240501 20:01:30.355154 30682 raft_consensus.cc:504] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:30.547127 29659 raft_consensus.cc:2379] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d in current term 2: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:30.546821 29698 leader_election.cc:401] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20240501 20:01:30.548568 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfac893b4b4f457694a4eb184b6abb53" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:30.549914 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d2cac25f354a97b1bfb53d57ae7c08" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:30.554853 30790 raft_consensus.cc:686] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:30.545687 29654 rpcz_store.cc:276] 0501 20:01:29.051105 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:30.331080 (+1279975us) service_pool.cc:225] Handling call
0501 20:01:30.331397 (+ 317us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:30.346995 (+ 15598us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:30.373491 (+ 26496us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:30.547672 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010ceabf3e5a4b1c9330b80dade765bb" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:30.516489 29808 raft_consensus.cc:2454] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
I20240501 20:01:30.559748 29805 raft_consensus.cc:2454] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 4.
I20240501 20:01:30.567142 29624 leader_election.cc:302] T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:30.531208 30817 leader_election.cc:288] T a8787bc4d6b44649b9138d906730110a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:30.602942 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5525fc4b82a44b0991c3dd6098fde81" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:30.603371 29700 leader_election.cc:302] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:30.603787 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af08384057a4d85977481472f460168" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:30.362344 29727 rpcz_store.cc:276] 0501 20:01:29.055561 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:29.091203 (+ 35642us) service_pool.cc:225] Handling call
0501 20:01:29.288330 (+197127us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:30.222534 (+934204us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:30.226299 (+ 3765us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:30.238196 (+ 11897us) raft_consensus.cc:1631] Marking committed up to 1
0501 20:01:30.240447 (+ 2251us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:30.244633 (+ 4186us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:30.270658 (+ 26025us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":521769,"raft.queue_time_us":619908,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":11287,"thread_start_us":363172,"threads_started":1}
I20240501 20:01:30.549552 30812 raft_consensus.cc:3037] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:30.574595 29730 raft_consensus.cc:2379] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 2: Already voted for candidate ee0d97247d394a8facfa774f15f3a67d in this term.
I20240501 20:01:30.579195 30829 leader_election.cc:288] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:30.495425 30834 raft_consensus.cc:504] T 1ca3948261374cef8b1b206a43c00173 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:30.545518 30839 raft_consensus.cc:482] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:30.552335 30820 leader_election.cc:288] T ec073fe645614fe28e7837de3c22510b P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:30.568439 30794 leader_election.cc:288] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:30.553709 29728 raft_consensus.cc:2454] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 3.
I20240501 20:01:30.569830 30805 raft_consensus.cc:482] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:30.602559 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48630ae6eb394ac1b54955dd01cf0ed2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:30.608270 29655 raft_consensus.cc:1261] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:01:30.529999 30762 consensus_queue.cc:260] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:30.539721 30798 consensus_queue.cc:237] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:30.602130 30828 raft_consensus.cc:504] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:30.616266 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcf30cf66714efdac997862dafbe4a2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:30.826742 30822 raft_consensus.cc:482] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:30.609987 30842 raft_consensus.cc:482] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:30.963238 29655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59136 (request call id 650) took 1868 ms. Trace:
I20240501 20:01:30.963398 30822 raft_consensus.cc:504] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:30.963699 29655 rpcz_store.cc:276] 0501 20:01:29.094946 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:30.364747 (+1269801us) service_pool.cc:225] Handling call
0501 20:01:30.365131 (+ 384us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:30.962987 (+597856us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:30.963188 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:30.964884 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3f29f0eeea4b119984634202e5d9cd" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:30.608004 30832 raft_consensus.cc:482] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:30.965999 30832 raft_consensus.cc:3037] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:30.963274 30842 raft_consensus.cc:504] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:30.851092 30768 leader_election.cc:288] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:30.967283 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1a08b455b94ae089396a15c837ddc2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:30.614131 30854 consensus_queue.cc:1035] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.355s
I20240501 20:01:30.907400 30847 tablet_replica.cc:330] T be383be783234855878cb8fd442c7b05 P 888d99522e4844118f8f1cf26f268c8c: stopping tablet replica
I20240501 20:01:30.970012 29735 raft_consensus.cc:2454] T 1ca3948261374cef8b1b206a43c00173 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
I20240501 20:01:30.970875 30829 raft_consensus.cc:2781] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:30.971320 30829 raft_consensus.cc:482] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240501 20:01:30.907171 30846 log.cc:924] Time spent T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d: Append to log took a long time: real 0.210s user 0.000s sys 0.000s
I20240501 20:01:30.971742 30829 raft_consensus.cc:3037] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:30.906370 29658 raft_consensus.cc:1261] T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:01:30.973404 29658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59136 (request call id 651) took 1878 ms. Trace:
I20240501 20:01:30.973547 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c651044a305c4168805825a53d910899" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:30.973774 29658 rpcz_store.cc:276] 0501 20:01:29.095301 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:30.382550 (+1287249us) service_pool.cc:225] Handling call
0501 20:01:30.450400 (+ 67850us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:30.973160 (+522760us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:30.973351 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:30.975303 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84fa28fb55f44b61ba1a88ed775a78ba" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:30.975340 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49da4246f5f6438abe0e7e8c9a523d80" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:30.548648 29651 rpcz_store.cc:276] 0501 20:01:29.314491 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:29.944162 (+629671us) service_pool.cc:225] Handling call
0501 20:01:30.455984 (+511822us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:30.976629 29651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668b4a4d782c4278932dd8413dc3ab3e" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:30.976307 30839 raft_consensus.cc:504] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:30.977267 29651 raft_consensus.cc:2454] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 2.
I20240501 20:01:30.978345 29651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca3948261374cef8b1b206a43c00173" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:30.979316 30839 leader_election.cc:288] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:30.979486 29651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a35893b91247a99912eba787d787cf" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:30.967913 29730 raft_consensus.cc:2454] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:30.609324 30844 consensus_queue.cc:1035] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.378s
I20240501 20:01:30.980280 29660 raft_consensus.cc:2454] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
I20240501 20:01:30.976207 29809 raft_consensus.cc:3037] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:30.688894 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211e27a3b0084b89a6dd7550849b5a65" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:30.981477 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9472c53f79ec423f82c396bd980674e2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:30.981900 29731 raft_consensus.cc:2454] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:31.013408 29800 raft_consensus.cc:2454] T d1d2cac25f354a97b1bfb53d57ae7c08 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
I20240501 20:01:30.982391 29729 raft_consensus.cc:2454] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:30.982797 30842 leader_election.cc:288] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:30.922118 30759 raft_consensus.cc:482] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:30.983557 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0374bb790ed345ed90452595f71383a2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:30.936805 30848 consensus_queue.cc:1035] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
W20240501 20:01:30.725167 30843 log.cc:924] Time spent T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.158s user 0.000s sys 0.000s
I20240501 20:01:30.654260 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f5d1686214485d91a0baf4b870740e" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:30.604068 30685 raft_consensus.cc:2781] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:30.620820 30809 leader_election.cc:288] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:30.696189 30840 raft_consensus.cc:3037] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 3
W20240501 20:01:30.683380 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 185701us
I20240501 20:01:30.690166 30761 leader_election.cc:288] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:30.996696 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211e27a3b0084b89a6dd7550849b5a65" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:30.987942 29809 raft_consensus.cc:2454] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 3.
W20240501 20:01:30.615993 29734 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 664) took 1990 ms (client timeout 1818 ms). Trace:
W20240501 20:01:30.970117 29728 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 652) took 2490 ms (client timeout 2418 ms). Trace:
I20240501 20:01:30.610004 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9046e38fc57f4364aa3d539e1fc01928" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:30.688220 30837 log.cc:924] Time spent T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 0.248s user 0.000s sys 0.001s
W20240501 20:01:30.709055 30845 log.cc:924] Time spent T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
W20240501 20:01:31.031875 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 428565us
I20240501 20:01:31.014796 30805 raft_consensus.cc:504] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:31.048352 29698 leader_election.cc:302] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:31.032327 29656 raft_consensus.cc:2362] T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 3. Re-sending same reply.
I20240501 20:01:30.688572 30790 consensus_queue.cc:237] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:30.652550 30650 raft_consensus.cc:504] T add9d7579351483fb268f89a44e95d8a P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:30.697347 29802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38cbeb0dc0c14e3d9182101eea885625" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:30.984846 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38cbeb0dc0c14e3d9182101eea885625" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:30.960454 30756 leader_election.cc:288] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:30.948292 30751 leader_election.cc:288] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:31.037142 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010ceabf3e5a4b1c9330b80dade765bb" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:31.015569 30822 leader_election.cc:288] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:30.687702 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8787bc4d6b44649b9138d906730110a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:31.056202 29731 raft_consensus.cc:2454] T 010ceabf3e5a4b1c9330b80dade765bb P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:31.057206 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcf30cf66714efdac997862dafbe4a2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:31.041779 29809 raft_consensus.cc:1261] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240501 20:01:30.631321 30815 leader_election.cc:288] T d1d2cac25f354a97b1bfb53d57ae7c08 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:31.018157 30820 raft_consensus.cc:482] T 456035a1cb3f4f7da52d0fc1dd1c656c P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:31.058887 29809 raft_consensus.cc:1261] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240501 20:01:31.040831 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d2cac25f354a97b1bfb53d57ae7c08" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:31.058610 30820 raft_consensus.cc:504] T 456035a1cb3f4f7da52d0fc1dd1c656c P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:31.060700 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac2171683884f6f97cdfeb0a656cb49" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:31.060925 30807 leader_election.cc:288] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:31.040884 29734 rpcz_store.cc:269] 0501 20:01:28.625809 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:28.644675 (+ 18866us) service_pool.cc:225] Handling call
0501 20:01:30.615846 (+1971171us) spinlock_profiling.cc:243] Waited 76 ms on lock 0x7b54001ef0d4. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787f4fe 00007fe45c7bd9d7 00007fe455ea1c14 00007fe455ea1ba2 00007fe455ea1ad1 00007fe455ea1a4c 00007fe455ea0d02 00007fe455ca2bcc 00007fe455ca2006 00007fe455ca19af
0501 20:01:30.615936 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":174820608}
I20240501 20:01:31.061367 29658 raft_consensus.cc:2454] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 2.
I20240501 20:01:31.062007 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668b4a4d782c4278932dd8413dc3ab3e" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:31.062371 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfac893b4b4f457694a4eb184b6abb53" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:31.062714 29734 raft_consensus.cc:2379] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 2: Already voted for candidate ee0d97247d394a8facfa774f15f3a67d in this term.
I20240501 20:01:31.063107 29658 raft_consensus.cc:2454] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 2.
I20240501 20:01:31.063505 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ed27033ab1645e9ab0600ab635d9589" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:31.055037 30856 raft_consensus.cc:482] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:31.064150 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1a08b455b94ae089396a15c837ddc2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
W20240501 20:01:31.064396 30740 consensus_meta.cc:287] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c: Time spent flushing consensus metadata: real 0.614s user 0.008s sys 0.001s
W20240501 20:01:31.042857 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 426968us
W20240501 20:01:31.045130 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 478002us
I20240501 20:01:31.066730 29624 leader_election.cc:302] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:31.045188 29800 raft_consensus.cc:1261] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240501 20:01:30.709198 30838 raft_consensus.cc:2781] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:30.957778 30787 raft_consensus.cc:504] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:31.034641 30685 raft_consensus.cc:482] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:30.615249 30748 raft_consensus.cc:686] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:31.092264 30685 raft_consensus.cc:3037] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:31.092808 30748 consensus_queue.cc:237] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:31.036736 30769 raft_consensus.cc:482] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:31.095811 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48fd0a8e289401aa7417c42ff2c81b2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:31.096082 30769 raft_consensus.cc:504] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:30.895865 30703 leader_election.cc:288] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:31.050655 29698 leader_election.cc:302] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
W20240501 20:01:31.051774 29656 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59142 (request call id 745) took 2246 ms (client timeout 2169 ms). Trace:
I20240501 20:01:31.034973 29804 raft_consensus.cc:2454] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 2.
I20240501 20:01:31.053632 29730 raft_consensus.cc:2379] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 2: Already voted for candidate ee0d97247d394a8facfa774f15f3a67d in this term.
I20240501 20:01:31.057206 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb27193abbf4644a48662909bc7b04a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:30.613793 30810 raft_consensus.cc:504] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:31.061090 30820 leader_election.cc:288] T 456035a1cb3f4f7da52d0fc1dd1c656c P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:30.960840 30814 leader_election.cc:288] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:31.026871 30762 raft_consensus.cc:3037] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:31.113749 29556 catalog_manager.cc:5484] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240501 20:01:31.078164 30753 consensus_meta.cc:287] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d: Time spent flushing consensus metadata: real 0.858s user 0.005s sys 0.005s
I20240501 20:01:31.118484 29556 catalog_manager.cc:5484] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:31.059865 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49da4246f5f6438abe0e7e8c9a523d80" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:31.120481 29660 raft_consensus.cc:2354] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d for earlier term 2. Current term is 3.
I20240501 20:01:31.120887 29653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59142 (request call id 747) took 2313 ms. Trace:
I20240501 20:01:31.074357 30756 raft_consensus.cc:2726] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:01:31.121215 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d88139f89c47ff84cad674147aae0f" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:31.076881 30839 raft_consensus.cc:2781] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240501 20:01:31.122299 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e89c0572e4ee7be8e5315b459bdb3" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:31.072036 29624 leader_election.cc:302] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:31.119896 29556 catalog_manager.cc:5484] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 1 to 3, leader changed from fdfeac4190e045f290e003b2298d988d (127.27.217.194) to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 3 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
W20240501 20:01:31.123379 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 51450.6us
I20240501 20:01:31.121963 29773 leader_election.cc:302] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 888d99522e4844118f8f1cf26f268c8c
I20240501 20:01:31.058928 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36057b3f24d44b9aac3a1378053a395f" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:31.031512 30759 raft_consensus.cc:504] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:31.047914 29654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48fd0a8e289401aa7417c42ff2c81b2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:31.052713 30645 leader_election.cc:288] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:31.076678 29800 raft_consensus.cc:1261] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240501 20:01:31.052067 30853 raft_consensus.cc:2781] T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:31.064517 30856 raft_consensus.cc:504] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:31.083357 29806 raft_consensus.cc:2454] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 2.
I20240501 20:01:31.085511 30805 leader_election.cc:288] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:31.097934 29656 rpcz_store.cc:269] 0501 20:01:28.802414 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:28.803865 (+ 1451us) service_pool.cc:225] Handling call
0501 20:01:30.654115 (+1850250us) spinlock_profiling.cc:243] Waited 2.62 ms on lock 0x7b5800001598. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe454797b51 00007fe4549193b1 00007fe45491972e 0000000000339c39 00007fe45c7486c0 00007fe45c80e3d4 00007fe45c80e441 00007fe45c7bdf60 00007fe45c7bd844 00007fe455ea1c14 00007fe455ea1ba2
0501 20:01:31.049211 (+395096us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6034688}
I20240501 20:01:31.100996 30769 leader_election.cc:288] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
W20240501 20:01:31.051934 29807 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41864 (request call id 760) took 2025 ms (client timeout 2247 ms). Trace:
I20240501 20:01:31.052366 30796 leader_election.cc:288] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:31.051671 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b337b8e3e38d4ad6b1076fbd0d0a1609" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:31.103677 30838 raft_consensus.cc:482] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:31.110788 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e38a2e91a3d48d09a22abce723af833" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
W20240501 20:01:31.111430 30384 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 2.03941e+06us
I20240501 20:01:31.112418 30754 leader_election.cc:288] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:31.087242 30829 raft_consensus.cc:504] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:31.113327 30650 leader_election.cc:288] T add9d7579351483fb268f89a44e95d8a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
W20240501 20:01:31.043432 29728 rpcz_store.cc:269] 0501 20:01:28.479263 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:28.479561 (+ 298us) service_pool.cc:225] Handling call
0501 20:01:29.952307 (+1472746us) spinlock_profiling.cc:243] Waited 8.68 ms on lock 0x7b58001b3098. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe454797b51 00007fe4549193b1 00007fe45491972e 0000000000339c39 00007fe45c7486c0 00007fe45c80e3d4 00007fe45c80e441 00007fe45c7bdf60 00007fe45c7bd844 00007fe455ea1c14 00007fe455ea1ba2
0501 20:01:30.970004 (+1017697us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":19969920}
I20240501 20:01:31.114917 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77be7101c0fe4085b6706296f609870a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:31.049356 30798 consensus_queue.cc:1046] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
I20240501 20:01:30.615536 30849 consensus_queue.cc:1035] T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.435s
I20240501 20:01:31.124022 29624 leader_election.cc:302] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:31.124313 29659 raft_consensus.cc:2454] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
I20240501 20:01:31.157378 29624 leader_election.cc:302] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:31.157904 29659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59136 (request call id 657) took 1961 ms. Trace:
I20240501 20:01:31.158190 29624 leader_election.cc:302] T 010ceabf3e5a4b1c9330b80dade765bb P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:31.158267 29659 rpcz_store.cc:276] 0501 20:01:29.196268 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:31.157210 (+1960942us) service_pool.cc:225] Handling call
0501 20:01:31.157477 (+ 267us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:31.157639 (+ 162us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:31.157662 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:31.157699 (+ 37us) raft_consensus.cc:1631] Marking committed up to 1
0501 20:01:31.157726 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:31.157773 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:31.157859 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:31.159066 30748 raft_consensus.cc:2781] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:31.159436 30748 raft_consensus.cc:482] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:31.159567 30856 leader_election.cc:288] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:31.127781 30817 consensus_queue.cc:1035] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20240501 20:01:31.160322 30769 raft_consensus.cc:2781] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:31.160686 30769 raft_consensus.cc:482] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:31.161067 30769 raft_consensus.cc:3037] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:31.064787 30740 raft_consensus.cc:504] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:31.121251 29653 rpcz_store.cc:276] 0501 20:01:28.802970 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:28.808145 (+ 5175us) service_pool.cc:225] Handling call
0501 20:01:29.329882 (+521737us) spinlock_profiling.cc:243] Waited 4.18 ms on lock 0x7b5800001598. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe454797b51 00007fe4549193b1 00007fe454919262 00007fe45d80b889 00007fe45c7483c0 00007fe45c80e3bb 00007fe45c80e441 00007fe45c7d2060 00007fe45c7bd5a9 00007fe45c7bd052 00007fe455e9f694
0501 20:01:29.330146 (+ 264us) raft_consensus.cc:1457] Updating replica for 1 ops
0501 20:01:29.330575 (+ 429us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:29.330612 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0501 20:01:29.955177 (+624565us) spinlock_profiling.cc:243] Waited 5.05 ms on lock 0x7b2c00042f48. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe45625df11 00007fe45626d542 00007fe4578b6fc1 00007fe45787b7da 00007fe45787abee 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc 00007fe455e9e782
0501 20:01:29.967563 (+ 12386us) log.cc:842] Serialized 26 byte log entry
0501 20:01:30.217585 (+250022us) raft_consensus.cc:1631] Marking committed up to 1
0501 20:01:30.217618 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:30.217687 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0501 20:01:31.115692 (+898005us) raft_consensus.cc:1671] finished
0501 20:01:31.115718 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:31.115884 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":22661,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":12513,"spinlock_wait_cycles":21227264,"thread_start_us":249713,"threads_started":1,"wal-append.queue_time_us":406475}
I20240501 20:01:31.145509 30761 raft_consensus.cc:2781] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:31.125591 30753 raft_consensus.cc:504] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:31.130406 29660 raft_consensus.cc:3037] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:01:31.133102 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 82523.5us
I20240501 20:01:31.174845 29698 leader_election.cc:302] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:31.175899 29698 leader_election.cc:302] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:31.141737 30815 consensus_queue.cc:1046] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:31.142184 30787 leader_election.cc:288] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:31.127741 30851 raft_consensus.cc:2781] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:31.128021 30842 raft_consensus.cc:2781] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:31.126145 30839 raft_consensus.cc:482] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:31.091056 30832 raft_consensus.cc:504] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:31.127933 30850 consensus_queue.cc:1035] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.534s
I20240501 20:01:31.150266 29654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59136 (request call id 656) took 1962 ms. Trace:
I20240501 20:01:31.191952 29654 rpcz_store.cc:276] 0501 20:01:29.187733 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:31.142001 (+1954268us) service_pool.cc:225] Handling call
0501 20:01:31.142334 (+ 333us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:31.142523 (+ 189us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:31.142550 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:31.142600 (+ 50us) raft_consensus.cc:1631] Marking committed up to 2
0501 20:01:31.144153 (+ 1553us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:31.146642 (+ 2489us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:31.150204 (+ 3562us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:31.193959 29654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59136 (request call id 659) took 1895 ms. Trace:
I20240501 20:01:31.194113 30832 leader_election.cc:288] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:31.194346 29654 rpcz_store.cc:276] 0501 20:01:29.298685 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:31.193095 (+1894410us) service_pool.cc:225] Handling call
0501 20:01:31.193423 (+ 328us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:31.193603 (+ 180us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:31.193639 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:31.193690 (+ 51us) raft_consensus.cc:1631] Marking committed up to 1
0501 20:01:31.193730 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:31.193792 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:31.193904 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:31.149089 30838 raft_consensus.cc:3037] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:01:31.196754 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 71574.6us
I20240501 20:01:31.150827 30834 leader_election.cc:288] T 1ca3948261374cef8b1b206a43c00173 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:31.129884 30853 raft_consensus.cc:482] T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:31.148169 29651 raft_consensus.cc:2379] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d in current term 3: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:31.155562 29730 raft_consensus.cc:2454] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:31.107579 30855 consensus_queue.cc:1035] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20240501 20:01:31.159843 30748 raft_consensus.cc:3037] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:31.185207 30851 raft_consensus.cc:482] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:31.159898 29659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59136 (request call id 658) took 1870 ms. Trace:
I20240501 20:01:31.169795 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ed27033ab1645e9ab0600ab635d9589" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:31.209419 29653 raft_consensus.cc:2454] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 888d99522e4844118f8f1cf26f268c8c in term 2.
I20240501 20:01:31.126688 29655 raft_consensus.cc:2454] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 2.
I20240501 20:01:31.171154 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8edcd903224d48f889b46e1b5318e0ec" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:31.180896 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcf30cf66714efdac997862dafbe4a2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:31.173933 30650 raft_consensus.cc:2781] T 010ceabf3e5a4b1c9330b80dade765bb P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:31.211352 29728 raft_consensus.cc:2454] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 2.
I20240501 20:01:31.172817 30740 leader_election.cc:288] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 3 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:31.174052 30761 raft_consensus.cc:482] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:31.183626 30859 raft_consensus.cc:2781] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:31.212270 30761 raft_consensus.cc:3037] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:31.212329 30859 raft_consensus.cc:482] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:31.212828 30859 raft_consensus.cc:3037] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:31.177248 29729 raft_consensus.cc:2379] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 2: Already voted for candidate ee0d97247d394a8facfa774f15f3a67d in this term.
I20240501 20:01:31.191282 30852 raft_consensus.cc:482] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:31.173187 30418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9046e38fc57f4364aa3d539e1fc01928" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c" is_pre_election: true
I20240501 20:01:31.198918 30853 raft_consensus.cc:3037] T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:31.200759 30685 raft_consensus.cc:504] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:31.180475 30794 raft_consensus.cc:2726] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:31.220417 29653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59136 (request call id 663) took 1807 ms. Trace:
I20240501 20:01:31.180536 30768 raft_consensus.cc:482] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:31.201782 30420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "696121417ea9403990456cfc6fc4f416" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c" is_pre_election: true
I20240501 20:01:31.226100 30420 raft_consensus.cc:2454] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 2.
I20240501 20:01:31.184144 30856 raft_consensus.cc:482] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:31.202883 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec073fe645614fe28e7837de3c22510b" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:31.227741 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69915dd5992641e3a24f7f45f1cd5c0c" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:31.228435 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36057b3f24d44b9aac3a1378053a395f" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:31.229076 29730 raft_consensus.cc:2454] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:31.203845 29624 leader_election.cc:302] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:31.123147 29700 leader_election.cc:302] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
W20240501 20:01:31.233685 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 110603us
I20240501 20:01:31.187947 30849 raft_consensus.cc:3032] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:01:31.205821 30753 leader_election.cc:288] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:31.205766 30800 leader_election.cc:288] T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:31.206563 30810 leader_election.cc:288] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:31.184618 30842 raft_consensus.cc:482] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:31.208792 30820 consensus_queue.cc:1035] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20240501 20:01:31.180410 30419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05df43bdfa479b902b1da297fb5974" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c" is_pre_election: true
I20240501 20:01:31.208593 29659 rpcz_store.cc:276] 0501 20:01:29.289668 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:31.159225 (+1869557us) service_pool.cc:225] Handling call
0501 20:01:31.159460 (+ 235us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:31.159609 (+ 149us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:31.159631 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:31.159667 (+ 36us) raft_consensus.cc:1631] Marking committed up to 1
0501 20:01:31.159693 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:31.159756 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:31.159853 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:31.209131 29654 raft_consensus.cc:3037] T 2d86f8848a184a87947fd927c44e5a35 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:31.211426 30650 raft_consensus.cc:482] T 010ceabf3e5a4b1c9330b80dade765bb P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:31.172128 30759 leader_election.cc:288] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:31.215627 29660 raft_consensus.cc:1261] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20240501 20:01:31.143397 29807 rpcz_store.cc:269] 0501 20:01:29.026879 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:29.041524 (+ 14645us) service_pool.cc:225] Handling call
0501 20:01:30.602122 (+1560598us) spinlock_profiling.cc:243] Waited 4.5 ms on lock 0x7b58000d2c98. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe454797b51 00007fe4549193b1 00007fe454919262 00007fe45d80b889 00007fe45c7483c0 00007fe45c80e3bb 00007fe45c80e441 00007fe45c7bdf60 00007fe45c7bd844 00007fe455ea1c14 00007fe455ea1ba2
0501 20:01:31.050015 (+447893us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10352384}
I20240501 20:01:31.221624 29653 rpcz_store.cc:276] 0501 20:01:29.413225 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:31.214437 (+1801212us) service_pool.cc:225] Handling call
0501 20:01:31.214718 (+ 281us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:31.215114 (+ 396us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:31.215154 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:31.215212 (+ 58us) raft_consensus.cc:1631] Marking committed up to 1
0501 20:01:31.220091 (+ 4879us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:31.220185 (+ 94us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:31.220341 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":377}
I20240501 20:01:31.223290 30383 leader_election.cc:302] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, 888d99522e4844118f8f1cf26f268c8c; no voters:
I20240501 20:01:31.209522 30851 raft_consensus.cc:3037] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:31.243103 29624 leader_election.cc:302] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:31.186836 30839 raft_consensus.cc:3037] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:31.246357 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5525f546426c47a38edf7f56ad04b7ab" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:31.497298 29731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 738) took 1029 ms. Trace:
I20240501 20:01:31.497700 29731 rpcz_store.cc:276] 0501 20:01:30.467784 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:31.246047 (+778263us) service_pool.cc:225] Handling call
0501 20:01:31.497239 (+251192us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:31.495088 29801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a35893b91247a99912eba787d787cf" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:31.498359 29659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59136 (request call id 664) took 1834 ms. Trace:
I20240501 20:01:31.498693 29801 raft_consensus.cc:2454] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
W20240501 20:01:31.499336 29801 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41864 (request call id 768) took 1464 ms (client timeout 1908 ms). Trace:
I20240501 20:01:31.500718 30685 leader_election.cc:288] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 election: Requested vote from peers ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
W20240501 20:01:31.499917 29801 rpcz_store.cc:269] 0501 20:01:30.035271 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:30.243136 (+207865us) service_pool.cc:225] Handling call
0501 20:01:31.494889 (+1251753us) spinlock_profiling.cc:243] Waited 2.69 ms on lock 0x7b58000d2c98. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe454797b51 00007fe4549193b1 00007fe454919262 00007fe45d80b889 00007fe45c7483c0 00007fe45c80e3bb 00007fe45c80e441 00007fe45c7bdf60 00007fe45c7bd844 00007fe455ea1c14 00007fe455ea1ba2
0501 20:01:31.499285 (+ 4396us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6177664}
W20240501 20:01:31.503123 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 260136us
I20240501 20:01:31.503860 29653 raft_consensus.cc:1226] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 2. Current term is 3. Ops: []
I20240501 20:01:31.504441 29653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59136 (request call id 665) took 1779 ms. Trace:
I20240501 20:01:31.504668 29653 rpcz_store.cc:276] 0501 20:01:29.729773 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:31.502755 (+1772982us) service_pool.cc:225] Handling call
0501 20:01:31.503058 (+ 303us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:31.504279 (+ 1221us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:31.504405 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:31.505108 29624 leader_election.cc:302] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:31.498857 29659 rpcz_store.cc:276] 0501 20:01:29.664363 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:31.497459 (+1833096us) service_pool.cc:225] Handling call
0501 20:01:31.497756 (+ 297us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:31.498061 (+ 305us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:31.498087 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:31.498132 (+ 45us) raft_consensus.cc:1631] Marking committed up to 1
0501 20:01:31.498160 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:31.498212 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:31.498310 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":78}
I20240501 20:01:31.506039 30849 raft_consensus.cc:729] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:31.311417 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "696121417ea9403990456cfc6fc4f416" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:31.506914 29728 raft_consensus.cc:2354] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 888d99522e4844118f8f1cf26f268c8c for earlier term 2. Current term is 3.
W20240501 20:01:31.505715 29657 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59136 (request call id 662) took 2127 ms (client timeout 1710 ms). Trace:
I20240501 20:01:31.502238 29660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59136 (request call id 654) took 2338 ms. Trace:
I20240501 20:01:31.312891 29734 raft_consensus.cc:2454] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 3.
W20240501 20:01:31.489007 30862 log.cc:924] Time spent T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 0.231s user 0.000s sys 0.001s
I20240501 20:01:31.222893 30840 raft_consensus.cc:2726] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959) with higher term. Message: Invalid argument: T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d for earlier term 2. Current term is 3.
I20240501 20:01:31.588141 29660 rpcz_store.cc:276] 0501 20:01:29.163449 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:31.130133 (+1966684us) service_pool.cc:225] Handling call
0501 20:01:31.130339 (+ 206us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:31.502002 (+371663us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:31.502189 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240501 20:01:31.281693 30863 log.cc:924] Time spent T 456035a1cb3f4f7da52d0fc1dd1c656c P ee0d97247d394a8facfa774f15f3a67d: Append to log took a long time: real 0.060s user 0.000s sys 0.001s
I20240501 20:01:31.559388 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31fff64b453a423eb0f567b3171564fb" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
W20240501 20:01:31.549060 29657 rpcz_store.cc:269] 0501 20:01:29.371173 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:29.805708 (+434535us) service_pool.cc:225] Handling call
0501 20:01:31.497930 (+1692222us) spinlock_profiling.cc:243] Waited 288 ms on lock 0x7b54000730d4. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787f4fe 00007fe45c7bd9d7 00007fe455ea1c14 00007fe455ea1ba2 00007fe455ea1ad1 00007fe455ea1a4c 00007fe455ea0d02 00007fe455ca2bcc 00007fe455ca2006 00007fe455ca19af
0501 20:01:31.498039 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":663109888}
W20240501 20:01:31.505653 30865 log.cc:924] Time spent T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 0.130s user 0.000s sys 0.000s
I20240501 20:01:31.590132 29700 leader_election.cc:302] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:31.267591 30794 raft_consensus.cc:2781] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240501 20:01:31.312912 30646 consensus_meta.cc:287] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f: Time spent flushing consensus metadata: real 0.786s user 0.009s sys 0.001s
I20240501 20:01:31.264196 30809 raft_consensus.cc:2781] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:31.507037 30849 consensus_queue.cc:260] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:31.641631 30809 raft_consensus.cc:482] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:31.641399 30646 raft_consensus.cc:504] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:31.642122 30809 raft_consensus.cc:3037] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:31.313840 30829 leader_election.cc:288] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 election: Requested vote from peers ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:31.628609 29730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 734) took 1162 ms. Trace:
I20240501 20:01:31.500380 30650 raft_consensus.cc:3037] T 010ceabf3e5a4b1c9330b80dade765bb P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:31.502451 30842 raft_consensus.cc:3037] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:31.224144 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e07220db7444ae97f294d70eda6a93" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:31.596194 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac2171683884f6f97cdfeb0a656cb49" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:31.264179 30768 raft_consensus.cc:504] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:31.629741 29657 raft_consensus.cc:1226] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 2. Current term is 3. Ops: []
I20240501 20:01:31.332197 30416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ed27033ab1645e9ab0600ab635d9589" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c" is_pre_election: true
I20240501 20:01:31.549350 30649 raft_consensus.cc:2781] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:31.596024 30419 raft_consensus.cc:2379] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d in current term 3: Already voted for candidate 888d99522e4844118f8f1cf26f268c8c in this term.
I20240501 20:01:31.549791 30800 raft_consensus.cc:2781] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:31.426105 30812 raft_consensus.cc:504] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:31.653065 30613 leader_election.cc:288] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:31.651525 29730 rpcz_store.cc:276] 0501 20:01:30.466664 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:31.559027 (+1092363us) service_pool.cc:225] Handling call
0501 20:01:31.628549 (+ 69522us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240501 20:01:31.589597 30383 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 371255us
I20240501 20:01:31.596499 30822 raft_consensus.cc:2781] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:31.665627 29731 raft_consensus.cc:2454] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 2.
I20240501 20:01:31.675271 29731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 755) took 1113 ms. Trace:
I20240501 20:01:31.633042 29801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "696121417ea9403990456cfc6fc4f416" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:31.675617 29731 rpcz_store.cc:276] 0501 20:01:30.561555 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:31.560153 (+998598us) service_pool.cc:225] Handling call
0501 20:01:31.675212 (+115059us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:31.264835 30834 raft_consensus.cc:482] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:31.628268 30794 raft_consensus.cc:482] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:31.507561 30838 raft_consensus.cc:504] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:31.668848 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e89c0572e4ee7be8e5315b459bdb3" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:31.669697 29656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59136 (request call id 655) took 2495 ms. Trace:
I20240501 20:01:31.669725 29657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59142 (request call id 763) took 1287 ms. Trace:
I20240501 20:01:31.647697 30748 raft_consensus.cc:504] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:31.494493 30859 raft_consensus.cc:504] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:31.329691 30415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e07220db7444ae97f294d70eda6a93" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c" is_pre_election: true
W20240501 20:01:31.672104 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 82032.6us
I20240501 20:01:31.320545 30769 raft_consensus.cc:504] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:31.673717 30800 raft_consensus.cc:482] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:31.681569 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49da4246f5f6438abe0e7e8c9a523d80" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:31.683375 29657 rpcz_store.cc:276] 0501 20:01:30.382099 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:31.629319 (+1247220us) service_pool.cc:225] Handling call
0501 20:01:31.629621 (+ 302us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:31.669525 (+ 39904us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:31.669682 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:31.694911 29657 raft_consensus.cc:1261] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240501 20:01:31.695708 29657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59142 (request call id 772) took 1267 ms. Trace:
I20240501 20:01:31.264266 30856 raft_consensus.cc:504] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:31.682873 29656 rpcz_store.cc:276] 0501 20:01:29.174896 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:31.131198 (+1956302us) service_pool.cc:225] Handling call
0501 20:01:31.131497 (+ 299us) raft_consensus.cc:1457] Updating replica for 1 ops
0501 20:01:31.131850 (+ 353us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:31.131882 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0501 20:01:31.132315 (+ 433us) log.cc:842] Serialized 26 byte log entry
0501 20:01:31.175341 (+ 43026us) raft_consensus.cc:1631] Marking committed up to 1
0501 20:01:31.175376 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:31.175439 (+ 63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0501 20:01:31.669486 (+494047us) raft_consensus.cc:1671] finished
0501 20:01:31.669506 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:31.669634 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":79,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":114,"thread_start_us":33788,"threads_started":1,"wal-append.queue_time_us":149851}
I20240501 20:01:31.691474 30416 raft_consensus.cc:2379] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 3: Already voted for candidate 888d99522e4844118f8f1cf26f268c8c in this term.
I20240501 20:01:31.670900 30649 raft_consensus.cc:482] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:31.690608 29700 leader_election.cc:302] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:31.702973 29730 raft_consensus.cc:3037] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:31.673460 30761 raft_consensus.cc:504] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:31.228679 30852 raft_consensus.cc:504] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:31.319253 30414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add9d7579351483fb268f89a44e95d8a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c" is_pre_election: true
I20240501 20:01:31.677385 30829 raft_consensus.cc:2781] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:01:31.705410 30822 raft_consensus.cc:482] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:31.712347 30829 raft_consensus.cc:686] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 2 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:31.712428 30822 raft_consensus.cc:3037] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:31.693147 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0374bb790ed345ed90452595f71383a2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:31.712314 29730 raft_consensus.cc:2454] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 3.
I20240501 20:01:31.713910 29731 raft_consensus.cc:2454] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 2.
I20240501 20:01:31.679121 30834 raft_consensus.cc:504] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } }
I20240501 20:01:31.680460 30740 raft_consensus.cc:482] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:31.696143 29657 rpcz_store.cc:276] 0501 20:01:30.427912 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:31.694213 (+1266301us) service_pool.cc:225] Handling call
0501 20:01:31.694512 (+ 299us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:31.695400 (+ 888us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:31.695640 (+ 240us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:31.697786 29656 raft_consensus.cc:1261] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Refusing update from remote peer 888d99522e4844118f8f1cf26f268c8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240501 20:01:31.685417 29653 raft_consensus.cc:1226] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 2. Current term is 3. Ops: []
I20240501 20:01:31.689724 29807 raft_consensus.cc:2454] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 888d99522e4844118f8f1cf26f268c8c in term 3.
I20240501 20:01:31.737211 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8edcd903224d48f889b46e1b5318e0ec" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:31.738507 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49da4246f5f6438abe0e7e8c9a523d80" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:31.738065 30822 raft_consensus.cc:504] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:31.739287 29807 raft_consensus.cc:2354] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d for earlier term 2. Current term is 3.
I20240501 20:01:31.702149 29726 raft_consensus.cc:2454] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 3.
I20240501 20:01:31.709342 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d86f8848a184a87947fd927c44e5a35" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:31.694507 30800 raft_consensus.cc:3037] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:31.711771 30761 leader_election.cc:288] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:31.714433 29731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 743) took 1190 ms. Trace:
I20240501 20:01:31.742394 29731 rpcz_store.cc:276] 0501 20:01:30.524411 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:31.676342 (+1151931us) service_pool.cc:225] Handling call
0501 20:01:31.714381 (+ 38039us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:31.680308 30794 raft_consensus.cc:3037] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:31.714778 29730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 737) took 1247 ms. Trace:
I20240501 20:01:31.720638 30838 leader_election.cc:288] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:31.721925 30768 leader_election.cc:288] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:31.705806 30649 raft_consensus.cc:3037] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:31.744751 29730 rpcz_store.cc:276] 0501 20:01:30.467510 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:31.681221 (+1213711us) service_pool.cc:225] Handling call
0501 20:01:31.714201 (+ 32980us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:31.753126 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ed27033ab1645e9ab0600ab635d9589" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:31.755079 29730 raft_consensus.cc:1261] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240501 20:01:31.709019 30762 raft_consensus.cc:2781] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:31.704123 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3f29f0eeea4b119984634202e5d9cd" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:31.749413 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6797d82dda2948d8a7445fcd33e44752" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:31.724419 30759 raft_consensus.cc:482] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:31.755774 29730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 741) took 1275 ms. Trace:
I20240501 20:01:31.763909 29730 rpcz_store.cc:276] 0501 20:01:30.480696 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:31.754355 (+1273659us) service_pool.cc:225] Handling call
0501 20:01:31.754670 (+ 315us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:31.755511 (+ 841us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:31.755711 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:31.761746 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a35893b91247a99912eba787d787cf" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:31.742416 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f261d2a05bf4c838532049601694b4a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:31.757220 30761 raft_consensus.cc:482] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:31.715169 30853 raft_consensus.cc:504] T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:31.789794 29801 raft_consensus.cc:2454] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 888d99522e4844118f8f1cf26f268c8c in term 2.
I20240501 20:01:31.791029 29801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0374bb790ed345ed90452595f71383a2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:31.743556 30832 raft_consensus.cc:482] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:31.765365 29729 raft_consensus.cc:2454] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 2.
I20240501 20:01:31.761018 30838 raft_consensus.cc:482] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:31.772173 30759 raft_consensus.cc:504] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:31.778764 29700 leader_election.cc:398] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Vote denied by peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861) with higher term. Message: Invalid argument: T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d for earlier term 2. Current term is 3.
I20240501 20:01:31.795580 29700 leader_election.cc:401] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240501 20:01:31.745317 29802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d88139f89c47ff84cad674147aae0f" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:31.781437 30851 raft_consensus.cc:504] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:31.757521 30762 raft_consensus.cc:482] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:31.713371 30829 consensus_queue.cc:237] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:31.761554 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77be7101c0fe4085b6706296f609870a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:31.791828 29801 raft_consensus.cc:2454] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 2.
I20240501 20:01:31.743783 30740 raft_consensus.cc:504] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:31.813473 29800 raft_consensus.cc:2454] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 2.
I20240501 20:01:31.814014 29801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8cd40cb4044d18a86001ce6ddf97ab" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:31.814309 29700 leader_election.cc:302] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:31.819816 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05df43bdfa479b902b1da297fb5974" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:31.820358 29801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca3948261374cef8b1b206a43c00173" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:31.823140 29700 leader_election.cc:302] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:31.762511 30385 leader_election.cc:302] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 888d99522e4844118f8f1cf26f268c8c, ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:31.808694 30762 raft_consensus.cc:3037] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Advancing to term 3
W20240501 20:01:31.826810 30385 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 79564.8us
I20240501 20:01:31.827750 29801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458740ab5c444b6ab3d1991c69db5084" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:31.828783 29801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8787bc4d6b44649b9138d906730110a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:31.810864 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668b4a4d782c4278932dd8413dc3ab3e" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:31.797659 30761 raft_consensus.cc:504] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:31.829741 30877 raft_consensus.cc:2781] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:01:31.821823 30685 raft_consensus.cc:2726] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861) with higher term. Message: Invalid argument: T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d for earlier term 2. Current term is 3.
I20240501 20:01:31.797364 30873 consensus_queue.cc:1035] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20240501 20:01:31.809950 30832 raft_consensus.cc:504] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:31.799371 30838 raft_consensus.cc:504] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:31.761389 30806 raft_consensus.cc:2932] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [term 2 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
I20240501 20:01:31.833575 30748 leader_election.cc:288] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:32.009583 30385 leader_election.cc:302] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 888d99522e4844118f8f1cf26f268c8c, ee0d97247d394a8facfa774f15f3a67d; no voters:
W20240501 20:01:31.829893 30867 log.cc:924] Time spent T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 0.162s user 0.001s sys 0.000s
I20240501 20:01:32.013567 29730 raft_consensus.cc:1261] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240501 20:01:32.015560 29801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "696121417ea9403990456cfc6fc4f416" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:32.017376 29801 raft_consensus.cc:3037] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Advancing to term 3
W20240501 20:01:31.885116 29735 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 685) took 1503 ms (client timeout 1776 ms). Trace:
I20240501 20:01:32.013247 30415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd5e83faaff4e2da4e4cc22ba410d33" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c" is_pre_election: true
I20240501 20:01:31.827562 30840 raft_consensus.cc:2781] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:31.810223 29802 raft_consensus.cc:2454] T f6d88139f89c47ff84cad674147aae0f P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
W20240501 20:01:32.017974 29735 rpcz_store.cc:269] 0501 20:01:30.381755 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:30.381891 (+ 136us) service_pool.cc:225] Handling call
0501 20:01:31.884964 (+1503073us) spinlock_profiling.cc:243] Waited 513 ms on lock 0x7b5400141254. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787f4fe 00007fe45c7bd9d7 00007fe455ea1c14 00007fe455ea1ba2 00007fe455ea1ad1 00007fe455ea1a4c 00007fe455ea0d02 00007fe455ca2bcc 00007fe455ca2006 00007fe455ca19af
0501 20:01:31.885064 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1179738368}
W20240501 20:01:31.823390 30383 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 86950.6us
I20240501 20:01:32.016083 30761 leader_election.cc:288] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:32.021984 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211e27a3b0084b89a6dd7550849b5a65" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:32.025071 29735 raft_consensus.cc:1226] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 2. Current term is 3. Ops: []
I20240501 20:01:32.026382 29735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 752) took 1465 ms. Trace:
I20240501 20:01:32.026973 29735 rpcz_store.cc:276] 0501 20:01:30.560730 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:32.023913 (+1463183us) service_pool.cc:225] Handling call
0501 20:01:32.024684 (+ 771us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:32.025863 (+ 1179us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:32.026296 (+ 433us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:31.836176 30812 leader_election.cc:288] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:31.851089 30859 leader_election.cc:288] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:31.852807 30852 leader_election.cc:288] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:31.871676 30768 raft_consensus.cc:482] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:31.921303 29654 raft_consensus.cc:1261] T 2d86f8848a184a87947fd927c44e5a35 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:01:31.918820 29729 raft_consensus.cc:1261] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240501 20:01:31.883793 30771 leader_election.cc:288] T c651044a305c4168805825a53d910899 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:31.847136 30769 leader_election.cc:288] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:32.006178 30834 leader_election.cc:288] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861), 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959)
W20240501 20:01:32.035604 29625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 173035us
I20240501 20:01:32.016438 30851 leader_election.cc:288] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:32.019310 30840 raft_consensus.cc:482] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:32.070633 30840 raft_consensus.cc:3037] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:32.019881 30759 leader_election.cc:288] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:32.038208 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd5e83faaff4e2da4e4cc22ba410d33" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:32.077739 29726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 732) took 1611 ms. Trace:
I20240501 20:01:32.040169 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec073fe645614fe28e7837de3c22510b" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:32.078226 29726 rpcz_store.cc:276] 0501 20:01:30.466081 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:30.549204 (+ 83123us) service_pool.cc:225] Handling call
0501 20:01:32.077527 (+1528323us) spinlock_profiling.cc:243] Waited 639 ms on lock 0x7b5400142654. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787f4fe 00007fe45c7bd9d7 00007fe455ea1c14 00007fe455ea1ba2 00007fe455ea1ad1 00007fe455ea1a4c 00007fe455ea0d02 00007fe455ca2bcc 00007fe455ca2006 00007fe455ca19af
0501 20:01:32.077667 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1469388160}
I20240501 20:01:31.851706 30817 raft_consensus.cc:482] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:32.078246 29656 raft_consensus.cc:2454] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 888d99522e4844118f8f1cf26f268c8c in term 4.
I20240501 20:01:32.080780 29726 raft_consensus.cc:1226] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:32.032635 30877 raft_consensus.cc:686] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [term 2 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:31.911227 30794 raft_consensus.cc:504] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:32.070215 30768 raft_consensus.cc:504] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:31.917812 30856 leader_election.cc:288] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:32.039814 30740 leader_election.cc:288] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:32.081198 30817 raft_consensus.cc:504] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:32.082036 30824 raft_consensus.cc:2781] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:01:32.036013 30828 leader_election.cc:288] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:32.089437 29653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59136 (request call id 666) took 2359 ms. Trace:
I20240501 20:01:32.115448 29653 rpcz_store.cc:276] 0501 20:01:29.730263 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:31.505211 (+1774948us) service_pool.cc:225] Handling call
0501 20:01:31.505401 (+ 190us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:31.735894 (+230493us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:32.089201 (+353307us) spinlock_profiling.cc:243] Waited 180 ms on lock 0x7b5400133b54. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787c1a1 00007fe45787abee 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc 00007fe455e9e782 00007fe455ca2bcc 00007fe455ca2006
0501 20:01:32.089380 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":413594496}
I20240501 20:01:32.091255 29726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 758) took 1528 ms. Trace:
W20240501 20:01:32.116945 29655 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59142 (request call id 761) took 1739 ms (client timeout 1781 ms). Trace:
I20240501 20:01:32.094292 29801 raft_consensus.cc:2454] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 3.
I20240501 20:01:32.077569 30827 leader_election.cc:288] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:32.071031 30878 raft_consensus.cc:2781] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:32.107132 30877 consensus_queue.cc:237] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:32.060106 30800 raft_consensus.cc:504] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:32.228770 29735 raft_consensus.cc:1261] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240501 20:01:32.232707 30817 leader_election.cc:288] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:32.233830 30836 consensus_queue.cc:260] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:32.235211 30836 raft_consensus.cc:3037] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:32.236688 29735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 756) took 1674 ms. Trace:
W20240501 20:01:32.117529 29655 rpcz_store.cc:269] 0501 20:01:30.377532 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:30.964642 (+587110us) service_pool.cc:225] Handling call
0501 20:01:32.116710 (+1152068us) spinlock_profiling.cc:243] Waited 161 ms on lock 0x7b5400132c54. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787f4fe 00007fe45c7bd9d7 00007fe455ea1c14 00007fe455ea1ba2 00007fe455ea1ad1 00007fe455ea1a4c 00007fe455ea0d02 00007fe455ca2bcc 00007fe455ca2006 00007fe455ca19af
0501 20:01:32.116857 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":370587520}
I20240501 20:01:32.237201 29735 rpcz_store.cc:276] 0501 20:01:30.561855 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:32.039159 (+1477304us) service_pool.cc:225] Handling call
0501 20:01:32.228298 (+189139us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:32.231890 (+ 3592us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:32.236614 (+ 4724us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240501 20:01:32.151276 29733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 679) took 3277 ms (client timeout 1710 ms). Trace:
I20240501 20:01:32.116781 29726 rpcz_store.cc:276] 0501 20:01:30.562396 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:32.079021 (+1516625us) service_pool.cc:225] Handling call
0501 20:01:32.079304 (+ 283us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:32.090937 (+ 11633us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:32.091178 (+ 241us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240501 20:01:32.239529 29733 rpcz_store.cc:269] 0501 20:01:28.873933 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:28.900270 (+ 26337us) service_pool.cc:225] Handling call
0501 20:01:32.151088 (+3250818us) spinlock_profiling.cc:243] Waited 1.39 s on lock 0x7b54003705d4. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787f4fe 00007fe45c7bd9d7 00007fe455ea1c14 00007fe455ea1ba2 00007fe455ea1ad1 00007fe455ea1a4c 00007fe455ea0d02 00007fe455ca2bcc 00007fe455ca2006 00007fe455ca19af
0501 20:01:32.151210 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3192135936}
I20240501 20:01:32.245323 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece8d4ed84c34b748f7b6272b31c90ac" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:32.245741 29733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 763) took 1083 ms. Trace:
I20240501 20:01:32.246258 29807 raft_consensus.cc:2454] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 3.
I20240501 20:01:32.097930 29729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 746) took 1530 ms. Trace:
I20240501 20:01:32.246251 29733 rpcz_store.cc:276] 0501 20:01:31.162568 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:32.245018 (+1082450us) service_pool.cc:225] Handling call
0501 20:01:32.245313 (+ 295us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:32.245468 (+ 155us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:32.245490 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:32.245523 (+ 33us) raft_consensus.cc:1631] Marking committed up to 1
0501 20:01:32.245549 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:32.245594 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:32.245691 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:32.250432 29733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 765) took 1087 ms. Trace:
I20240501 20:01:32.250875 29733 rpcz_store.cc:276] 0501 20:01:31.163229 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:32.249389 (+1086160us) service_pool.cc:225] Handling call
0501 20:01:32.249695 (+ 306us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:32.250136 (+ 441us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:32.250164 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:32.250212 (+ 48us) raft_consensus.cc:1631] Marking committed up to 1
0501 20:01:32.250237 (+ 25us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:32.250283 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:32.250386 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:32.098193 29654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59136 (request call id 661) took 2762 ms. Trace:
W20240501 20:01:32.241874 30809 consensus_meta.cc:287] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f: Time spent flushing consensus metadata: real 0.599s user 0.009s sys 0.003s
I20240501 20:01:32.260298 29726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 762) took 1099 ms. Trace:
W20240501 20:01:32.264037 30650 consensus_meta.cc:287] T 010ceabf3e5a4b1c9330b80dade765bb P 4c8da1d580614449a636b855bf11b95f: Time spent flushing consensus metadata: real 0.598s user 0.009s sys 0.001s
I20240501 20:01:32.264592 29735 raft_consensus.cc:1226] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:32.265316 29726 rpcz_store.cc:276] 0501 20:01:31.160546 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:32.240529 (+1079983us) service_pool.cc:225] Handling call
0501 20:01:32.259825 (+ 19296us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:32.259968 (+ 143us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:32.259991 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:32.260024 (+ 33us) raft_consensus.cc:1631] Marking committed up to 1
0501 20:01:32.260095 (+ 71us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:32.260139 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:32.260236 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:32.227660 30851 raft_consensus.cc:482] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:32.267692 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d86f8848a184a87947fd927c44e5a35" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:32.159344 30832 leader_election.cc:288] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:32.210045 30794 leader_election.cc:288] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:32.267838 29735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 760) took 1704 ms. Trace:
W20240501 20:01:32.111690 30385 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 104896us
I20240501 20:01:32.138357 30824 raft_consensus.cc:686] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [term 2 LEADER]: Becoming Leader. State: Replica: 888d99522e4844118f8f1cf26f268c8c, State: Running, Role: LEADER
I20240501 20:01:32.236815 30838 leader_election.cc:288] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:32.269567 29735 rpcz_store.cc:276] 0501 20:01:30.562926 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:32.252119 (+1689193us) service_pool.cc:225] Handling call
0501 20:01:32.264273 (+ 12154us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:32.266382 (+ 2109us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:32.267783 (+ 1401us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:32.253652 30852 raft_consensus.cc:482] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:32.272754 29726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 770) took 1106 ms. Trace:
I20240501 20:01:32.146232 30840 raft_consensus.cc:504] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:32.273591 29726 rpcz_store.cc:276] 0501 20:01:31.166022 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:32.266396 (+1100374us) service_pool.cc:225] Handling call
0501 20:01:32.272147 (+ 5751us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:32.272307 (+ 160us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:32.272331 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:32.272527 (+ 196us) raft_consensus.cc:1631] Marking committed up to 1
0501 20:01:32.272559 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:32.272611 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:32.272710 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:32.152621 30383 leader_election.cc:302] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, 888d99522e4844118f8f1cf26f268c8c; no voters:
I20240501 20:01:32.276028 30852 raft_consensus.cc:504] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:32.279393 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb27193abbf4644a48662909bc7b04a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:32.250136 29729 rpcz_store.cc:276] 0501 20:01:30.559044 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:31.817071 (+1258027us) service_pool.cc:225] Handling call
0501 20:01:31.844884 (+ 27813us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:32.064489 (+219605us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:32.089676 (+ 25187us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:32.267192 30650 raft_consensus.cc:504] T 010ceabf3e5a4b1c9330b80dade765bb P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:32.270344 30851 raft_consensus.cc:504] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:32.267035 30809 raft_consensus.cc:504] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:32.284629 29700 leader_election.cc:302] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:32.286499 29660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59142 (request call id 760) took 1909 ms. Trace:
W20240501 20:01:32.282157 30383 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 134388us
I20240501 20:01:32.312476 29557 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33356 (request call id 49) took 4430 ms. Trace:
I20240501 20:01:32.315374 29556 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:43370 (request call id 153) took 3899 ms. Trace:
I20240501 20:01:32.303093 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b337b8e3e38d4ad6b1076fbd0d0a1609" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:32.303943 29732 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 688) took 1919 ms (client timeout 2238 ms). Trace:
I20240501 20:01:32.284210 29807 raft_consensus.cc:2396] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20240501 20:01:32.363556 29803 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41858 (request call id 742) took 1915 ms (client timeout 1775 ms). Trace:
W20240501 20:01:32.372241 29803 rpcz_store.cc:269] 0501 20:01:30.448689 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:30.532767 (+ 84078us) service_pool.cc:225] Handling call
0501 20:01:32.302908 (+1770141us) spinlock_profiling.cc:243] Waited 8.34 ms on lock 0x7b58000d2c98. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe454797b51 00007fe4549193b1 00007fe454919262 00007fe45d80b889 00007fe45c7483c0 00007fe45c80e3bb 00007fe45c80e441 00007fe45c7bdf60 00007fe45c7bd844 00007fe455ea1c14 00007fe455ea1ba2
0501 20:01:32.363509 (+ 60601us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":19178880}
I20240501 20:01:32.260689 29654 rpcz_store.cc:276] 0501 20:01:29.336283 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:31.195414 (+1859131us) service_pool.cc:225] Handling call
0501 20:01:31.207998 (+ 12584us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:32.064458 (+856460us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:32.089584 (+ 25126us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240501 20:01:32.363273 29732 rpcz_store.cc:269] 0501 20:01:30.384116 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:30.384219 (+ 103us) service_pool.cc:225] Handling call
0501 20:01:32.303718 (+1919499us) spinlock_profiling.cc:243] Waited 59.7 ms on lock 0x7b5400371254. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787f4fe 00007fe45c7bd9d7 00007fe455ea1c14 00007fe455ea1ba2 00007fe455ea1ad1 00007fe455ea1a4c 00007fe455ea0d02 00007fe455ca2bcc 00007fe455ca2006 00007fe455ca19af
0501 20:01:32.303865 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":137271040}
W20240501 20:01:32.361928 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 77381.7us
I20240501 20:01:32.362387 29557 rpcz_store.cc:276] 0501 20:01:27.882085 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:27.882272 (+ 187us) service_pool.cc:225] Handling call
0501 20:01:29.790090 (+1907818us) spinlock_profiling.cc:243] Waited 691 ms on lock 0x7b54000037d4. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe457800d30 00007fe45788417a 00007fe45d7d5b1c 00007fe45d8cfa8e 00007fe45bbe3a44 00007fe45bbe39d2 00007fe45bbe3901 00007fe45bbe387c 00007fe45bbe2b32 00007fe455ca2bcc 00007fe455ca2006
0501 20:01:32.312393 (+2522303us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":146784,"reported_tablets":1,"spinlock_wait_cycles":1589056768}
W20240501 20:01:32.302116 30842 consensus_meta.cc:287] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d: Time spent flushing consensus metadata: real 0.633s user 0.008s sys 0.002s
W20240501 20:01:32.285154 30839 consensus_meta.cc:287] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f: Time spent flushing consensus metadata: real 0.700s user 0.010s sys 0.002s
I20240501 20:01:32.361652 29660 rpcz_store.cc:276] 0501 20:01:30.377259 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:31.595101 (+1217842us) service_pool.cc:225] Handling call
0501 20:01:31.595406 (+ 305us) raft_consensus.cc:1457] Updating replica for 1 ops
0501 20:01:31.675521 (+ 80115us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:31.675549 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0501 20:01:31.679615 (+ 4066us) log.cc:842] Serialized 26 byte log entry
0501 20:01:31.743064 (+ 63449us) raft_consensus.cc:1631] Marking committed up to 1
0501 20:01:31.743099 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:32.286126 (+543027us) spinlock_profiling.cc:243] Waited 79.7 ms on lock 0x7b54001a22d4. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787c1a1 00007fe45787abee 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc 00007fe455e9e782 00007fe455ca2bcc 00007fe455ca2006
0501 20:01:32.286169 (+ 43us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0501 20:01:32.286225 (+ 56us) raft_consensus.cc:1671] finished
0501 20:01:32.286238 (+ 13us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:32.286442 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":73,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":5860,"spinlock_wait_cycles":183376768,"thread_start_us":49555,"threads_started":1,"wal-append.queue_time_us":143895}
I20240501 20:01:32.281041 30768 leader_election.cc:288] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:32.411101 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458740ab5c444b6ab3d1991c69db5084" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:32.376502 29700 leader_election.cc:302] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
W20240501 20:01:32.287029 30649 consensus_meta.cc:287] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f: Time spent flushing consensus metadata: real 0.522s user 0.008s sys 0.002s
I20240501 20:01:32.309216 30646 leader_election.cc:288] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:32.297231 30840 leader_election.cc:288] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:32.422137 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211e27a3b0084b89a6dd7550849b5a65" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:32.284032 30884 raft_consensus.cc:2781] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240501 20:01:32.308849 30417 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40670 (request call id 36) took 1104 ms. Trace:
I20240501 20:01:32.316810 30878 raft_consensus.cc:482] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:32.362787 30886 raft_consensus.cc:2781] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:32.304328 30852 leader_election.cc:288] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:32.294216 30800 leader_election.cc:288] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:32.302579 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d86f8848a184a87947fd927c44e5a35" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
W20240501 20:01:32.285694 29651 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59142 (request call id 770) took 1897 ms (client timeout 1908 ms). Trace:
W20240501 20:01:32.444072 29651 rpcz_store.cc:269] 0501 20:01:30.387976 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:30.979228 (+591252us) service_pool.cc:225] Handling call
0501 20:01:32.285478 (+1306250us) spinlock_profiling.cc:243] Waited 105 ms on lock 0x7b54001c3254. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787f4fe 00007fe45c7bd9d7 00007fe455ea1c14 00007fe455ea1ba2 00007fe455ea1ad1 00007fe455ea1a4c 00007fe455ea0d02 00007fe455ca2bcc 00007fe455ca2006 00007fe455ca19af
0501 20:01:32.285635 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":241625856}
W20240501 20:01:32.430001 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 53600us
I20240501 20:01:32.439550 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add9d7579351483fb268f89a44e95d8a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:32.447552 29700 leader_election.cc:302] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:32.310671 30762 raft_consensus.cc:504] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:32.453871 29735 raft_consensus.cc:1261] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Refusing update from remote peer 888d99522e4844118f8f1cf26f268c8c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20240501 20:01:32.454360 29735 pending_rounds.cc:76] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d: Aborting all ops after (but not including) 4
I20240501 20:01:32.456372 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af08384057a4d85977481472f460168" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:32.459070 30840 raft_consensus.cc:2781] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:32.459645 30840 raft_consensus.cc:482] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:32.460100 30840 raft_consensus.cc:3037] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:32.463382 30849 raft_consensus.cc:3037] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:32.426146 30884 raft_consensus.cc:482] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [term 4 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:32.442651 30794 raft_consensus.cc:2781] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:01:32.441479 30878 raft_consensus.cc:3037] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:32.454620 29735 pending_rounds.cc:97] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20240501 20:01:32.472560 29735 raft_consensus.cc:2866] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 20:01:32.416929 30842 raft_consensus.cc:504] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:32.474906 29731 raft_consensus.cc:2379] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 3: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:32.424394 30649 raft_consensus.cc:504] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:32.480567 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77be7101c0fe4085b6706296f609870a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:32.441679 30886 raft_consensus.cc:482] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:32.422298 30839 raft_consensus.cc:504] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:32.469789 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f5d1686214485d91a0baf4b870740e" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:32.485549 29803 raft_consensus.cc:2454] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 3.
I20240501 20:01:32.362576 29556 rpcz_store.cc:276] 0501 20:01:28.462935 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:28.470059 (+ 7124us) service_pool.cc:225] Handling call
0501 20:01:28.752454 (+282395us) spinlock_profiling.cc:243] Waited 96.6 ms on lock 0x7b54000037d4. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe457800d30 00007fe45788417a 00007fe45d7d5b1c 00007fe45d8cfa8e 00007fe45bbe3a44 00007fe45bbe39d2 00007fe45bbe3901 00007fe45bbe387c 00007fe45bbe2b32 00007fe455ca2bcc 00007fe455ca2006
0501 20:01:32.315266 (+3562812us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0501 20:01:31.180305 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 20:01:31.221967 (+ 41662us) write_op.cc:424] Acquiring schema lock in shared mode
0501 20:01:31.224353 (+ 2386us) write_op.cc:427] Acquired schema lock
0501 20:01:31.228205 (+ 3852us) tablet.cc:660] Decoding operations
0501 20:01:31.332488 (+104283us) write_op.cc:612] Acquiring the partition lock for write op
0501 20:01:31.332552 (+ 64us) write_op.cc:633] Partition lock acquired for write op
0501 20:01:31.332592 (+ 40us) tablet.cc:683] Acquiring locks for 3 operations
0501 20:01:31.332965 (+ 373us) tablet.cc:699] Row locks acquired
0501 20:01:31.332986 (+ 21us) write_op.cc:260] PREPARE: finished
0501 20:01:31.333145 (+ 159us) write_op.cc:270] Start()
0501 20:01:31.333264 (+ 119us) write_op.cc:276] Timestamp: P: 1714593691333097 usec, L: 0
0501 20:01:31.333288 (+ 24us) op_driver.cc:331] REPLICATION: starting
0501 20:01:31.334049 (+ 761us) log.cc:842] Serialized 1320 byte log entry
0501 20:01:31.685353 (+351304us) op_driver.cc:444] REPLICATION: finished
0501 20:01:32.033760 (+348407us) write_op.cc:301] APPLY: starting
0501 20:01:32.033843 (+ 83us) tablet.cc:1365] starting BulkCheckPresence
0501 20:01:32.033974 (+ 131us) tablet.cc:1368] finished BulkCheckPresence
0501 20:01:32.034104 (+ 130us) tablet.cc:1370] starting ApplyRowOperation cycle
0501 20:01:32.070247 (+ 36143us) tablet.cc:1381] finished ApplyRowOperation cycle
0501 20:01:32.070501 (+ 254us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0501 20:01:32.070578 (+ 77us) write_op.cc:312] APPLY: finished
0501 20:01:32.070866 (+ 288us) log.cc:842] Serialized 34 byte log entry
0501 20:01:32.071052 (+ 186us) write_op.cc:481] Releasing partition, row and schema locks
0501 20:01:32.071175 (+ 123us) write_op.cc:446] Released schema lock
0501 20:01:32.071244 (+ 69us) write_op.cc:341] FINISH: Updating metrics
0501 20:01:32.071576 (+ 332us) write_op.cc:481] Releasing partition, row and schema locks
0501 20:01:32.071629 (+ 53us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":221758,"reported_tablets":3,"spinlock_wait_cycles":222182400,"child_traces":[["op",{"apply.queue_time_us":336174,"apply.run_cpu_time_us":3368,"apply.run_wall_time_us":41025,"num_ops":3,"prepare.queue_time_us":25613,"prepare.run_cpu_time_us":15770,"prepare.run_wall_time_us":414215,"raft.queue_time_us":157189,"raft.run_cpu_time_us":69594,"raft.run_wall_time_us":150255,"replication_time_us":345398,"thread_start_us":425349,"threads_started":4,"wal-append.queue_time_us":169198}]]}
I20240501 20:01:32.474323 29727 raft_consensus.cc:2379] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 2: Already voted for candidate ee0d97247d394a8facfa774f15f3a67d in this term.
I20240501 20:01:32.495469 29800 raft_consensus.cc:3037] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Advancing to term 3
W20240501 20:01:32.506129 30883 log.cc:924] Time spent T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c: Append to log took a long time: real 0.095s user 0.000s sys 0.000s
I20240501 20:01:32.507404 30418 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40658 (request call id 61) took 1284 ms. Trace:
I20240501 20:01:32.507810 30418 rpcz_store.cc:276] 0501 20:01:31.223418 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:31.228938 (+ 5520us) service_pool.cc:225] Handling call
0501 20:01:31.315505 (+ 86567us) raft_consensus.cc:1457] Updating replica for 1 ops
0501 20:01:31.689999 (+374494us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:31.692610 (+ 2611us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0501 20:01:31.761893 (+ 69283us) log.cc:842] Serialized 26 byte log entry
0501 20:01:32.069551 (+307658us) raft_consensus.cc:1631] Marking committed up to 4
0501 20:01:32.069613 (+ 62us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:32.069715 (+ 102us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0501 20:01:32.507119 (+437404us) raft_consensus.cc:1671] finished
0501 20:01:32.507140 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:32.507328 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26059,"raft.queue_time_us":105876,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":3916,"thread_start_us":343960,"threads_started":2,"wal-append.queue_time_us":502701}
I20240501 20:01:32.473299 30761 raft_consensus.cc:482] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:32.471201 30884 raft_consensus.cc:3037] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:32.425912 30824 consensus_queue.cc:237] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:32.518474 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "456035a1cb3f4f7da52d0fc1dd1c656c" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:32.542659 29658 raft_consensus.cc:2354] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d for earlier term 4. Current term is 5.
I20240501 20:01:32.481236 30794 raft_consensus.cc:686] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [term 3 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:32.557080 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c597f15e844f30837012baed4b0501" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:32.561168 30869 log.cc:924] Time spent T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d: Append to log took a long time: real 0.882s user 0.000s sys 0.004s
I20240501 20:01:32.477733 30851 leader_election.cc:288] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:32.484846 30886 raft_consensus.cc:3037] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:32.511175 30853 leader_election.cc:288] T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:32.562914 29803 raft_consensus.cc:2454] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 3.
I20240501 20:01:32.564599 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211e27a3b0084b89a6dd7550849b5a65" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:32.564782 29800 raft_consensus.cc:2454] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 3.
W20240501 20:01:32.567490 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 1.38844e+06us
I20240501 20:01:32.567960 30829 raft_consensus.cc:482] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:32.568465 29698 leader_election.cc:302] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:32.569202 29700 leader_election.cc:302] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
W20240501 20:01:32.539891 29727 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 754) took 1976 ms (client timeout 1999 ms). Trace:
I20240501 20:01:32.572799 29698 leader_election.cc:302] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:32.577791 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfac893b4b4f457694a4eb184b6abb53" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:32.531167 30761 raft_consensus.cc:504] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:32.576543 30836 raft_consensus.cc:2781] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:32.587579 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1a08b455b94ae089396a15c837ddc2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:32.638787 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9046e38fc57f4364aa3d539e1fc01928" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:32.430006 30417 rpcz_store.cc:276] 0501 20:01:31.217818 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:31.223251 (+ 5433us) service_pool.cc:225] Handling call
0501 20:01:31.223622 (+ 371us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:31.745829 (+522207us) raft_consensus.cc:1520] Early marking committed up to index 6
0501 20:01:31.859714 (+113885us) log.cc:842] Serialized 14 byte log entry
0501 20:01:32.252740 (+393026us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:32.254938 (+ 2198us) raft_consensus.cc:1631] Marking committed up to 6
0501 20:01:32.257990 (+ 3052us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:32.308621 (+ 50631us) spinlock_profiling.cc:243] Waited 491 ms on lock 0x7b5400611554. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787c1a1 00007fe45787abee 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc 00007fe455e9e782 00007fe455ca2bcc 00007fe455ca2006
0501 20:01:32.308676 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:32.308793 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":320569,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":3078,"raft.queue_time_us":125622,"raft.run_wall_time_us":13506,"spinlock_wait_cycles":1150645888,"thread_start_us":304241,"threads_started":3,"wal-append.queue_time_us":152934}
I20240501 20:01:32.572274 30762 leader_election.cc:288] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:32.652756 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 80007.4us
I20240501 20:01:32.648316 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a35893b91247a99912eba787d787cf" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:32.653439 29803 raft_consensus.cc:3037] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 3
I20240501 20:01:32.654520 29698 leader_election.cc:302] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
W20240501 20:01:32.655668 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 78582.4us
I20240501 20:01:32.655892 29698 leader_election.cc:398] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Vote denied by peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959) with higher term. Message: Invalid argument: T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d for earlier term 2. Current term is 3.
I20240501 20:01:32.657336 29698 leader_election.cc:302] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:32.662673 29625 leader_election.cc:302] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:32.663847 29625 leader_election.cc:302] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:32.666391 30414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e89c0572e4ee7be8e5315b459bdb3" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c" is_pre_election: true
I20240501 20:01:32.666718 30415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add9d7579351483fb268f89a44e95d8a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c" is_pre_election: true
I20240501 20:01:32.667872 30851 raft_consensus.cc:2781] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election won for term 3
W20240501 20:01:32.570739 29727 rpcz_store.cc:269] 0501 20:01:30.561285 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:32.302167 (+1740882us) service_pool.cc:225] Handling call
0501 20:01:32.537903 (+235736us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:32.668383 30851 raft_consensus.cc:482] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:32.668797 30851 raft_consensus.cc:3037] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:32.587769 30759 raft_consensus.cc:2781] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:32.588523 30836 raft_consensus.cc:482] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:32.597766 30650 leader_election.cc:288] T 010ceabf3e5a4b1c9330b80dade765bb P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:32.562333 30840 raft_consensus.cc:504] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:32.673729 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be383be783234855878cb8fd442c7b05" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:32.593294 30842 leader_election.cc:288] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:32.670812 30853 raft_consensus.cc:2726] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240501 20:01:32.568502 30829 raft_consensus.cc:504] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:32.678082 29625 leader_election.cc:302] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:32.677127 30839 leader_election.cc:288] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:32.668401 30762 raft_consensus.cc:482] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:32.654760 30877 raft_consensus.cc:2781] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:32.671805 30759 raft_consensus.cc:482] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:32.679571 30815 raft_consensus.cc:482] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:32.677830 30836 raft_consensus.cc:3037] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:32.684397 30650 consensus_queue.cc:1035] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:01:32.689492 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38cbeb0dc0c14e3d9182101eea885625" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:32.680146 30832 raft_consensus.cc:2726] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:32.686519 30806 raft_consensus.cc:2781] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:32.686877 30800 raft_consensus.cc:2781] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:01:32.609519 30794 consensus_queue.cc:237] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:32.693646 30878 raft_consensus.cc:504] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:32.702817 29803 raft_consensus.cc:2454] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 3.
I20240501 20:01:32.703873 30800 raft_consensus.cc:686] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:32.691236 30759 raft_consensus.cc:3037] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:32.704701 30414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05df43bdfa479b902b1da297fb5974" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c"
I20240501 20:01:32.704828 30800 consensus_queue.cc:237] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:32.705560 30414 raft_consensus.cc:2354] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d for earlier term 2. Current term is 3.
I20240501 20:01:32.706895 30847 raft_consensus.cc:2229] T be383be783234855878cb8fd442c7b05 P 888d99522e4844118f8f1cf26f268c8c [term 2 LEARNER]: Raft consensus shutting down.
I20240501 20:01:32.707404 30847 pending_rounds.cc:62] T be383be783234855878cb8fd442c7b05 P 888d99522e4844118f8f1cf26f268c8c: Trying to abort 1 pending ops.
I20240501 20:01:32.707706 30847 pending_rounds.cc:68] T be383be783234855878cb8fd442c7b05 P 888d99522e4844118f8f1cf26f268c8c: Aborting op as it isn't in flight: id { term: 2 index: 3 } timestamp: 7022975741085065216 op_type: NO_OP noop_request { }
I20240501 20:01:32.708233 30847 raft_consensus.cc:2866] T be383be783234855878cb8fd442c7b05 P 888d99522e4844118f8f1cf26f268c8c [term 2 LEARNER]: NO_OP replication failed: Aborted: Op aborted
I20240501 20:01:32.708688 30847 raft_consensus.cc:2258] T be383be783234855878cb8fd442c7b05 P 888d99522e4844118f8f1cf26f268c8c [term 2 LEARNER]: Raft consensus is shut down!
I20240501 20:01:32.696264 30815 raft_consensus.cc:504] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:32.710389 30815 leader_election.cc:288] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:32.696379 30844 raft_consensus.cc:482] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:32.711474 30847 ts_tablet_manager.cc:1875] T be383be783234855878cb8fd442c7b05 P 888d99522e4844118f8f1cf26f268c8c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:01:32.711218 30844 raft_consensus.cc:504] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:32.700896 30646 consensus_queue.cc:1035] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
I20240501 20:01:32.713501 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48630ae6eb394ac1b54955dd01cf0ed2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:32.695842 29625 leader_election.cc:302] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:32.688895 30877 raft_consensus.cc:482] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting leader election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:32.713146 29700 leader_election.cc:302] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:32.701298 30806 raft_consensus.cc:482] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:32.717880 30806 raft_consensus.cc:3037] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:01:32.696000 29772 connection.cc:232] RPC call timeout handler was delayed by -1.3039531490001082s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1534s for a response
W20240501 20:01:32.721172 29772 connection.cc:232] RPC call timeout handler was delayed by -1.1491531490000853s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1706s for a response
W20240501 20:01:32.721666 29772 connection.cc:232] RPC call timeout handler was delayed by -1.0861531489999834s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1776s for a response
W20240501 20:01:32.722071 29772 connection.cc:232] RPC call timeout handler was delayed by -1.0033531490000769s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18680000000000002s for a response
W20240501 20:01:32.724756 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 67524.4us
I20240501 20:01:32.727253 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1a08b455b94ae089396a15c837ddc2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:32.719928 30877 raft_consensus.cc:3037] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:32.729249 30418 raft_consensus.cc:1226] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 4 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 3. Current term is 4. Ops: []
I20240501 20:01:32.730854 30897 consensus_queue.cc:1046] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:32.731755 30897 raft_consensus.cc:3032] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:01:32.726326 30646 raft_consensus.cc:2781] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:32.732462 30897 raft_consensus.cc:729] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:32.732725 30646 raft_consensus.cc:482] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:32.733204 30646 raft_consensus.cc:3037] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:32.733520 30897 consensus_queue.cc:260] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:32.734527 30829 leader_election.cc:288] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:32.736402 30897 raft_consensus.cc:3037] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:32.744321 30844 leader_election.cc:288] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:32.726737 30832 raft_consensus.cc:2781] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:01:32.745638 29806 raft_consensus.cc:2396] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240501 20:01:32.745774 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e38a2e91a3d48d09a22abce723af833" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:32.747081 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48630ae6eb394ac1b54955dd01cf0ed2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:32.747404 30878 leader_election.cc:288] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 4 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:32.747695 30815 raft_consensus.cc:482] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:32.748252 30815 raft_consensus.cc:504] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:32.694929 30839 consensus_queue.cc:1035] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
I20240501 20:01:32.749356 29730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 745) took 2190 ms. Trace:
I20240501 20:01:32.749717 29730 rpcz_store.cc:276] 0501 20:01:30.558771 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:31.764752 (+1205981us) service_pool.cc:225] Handling call
0501 20:01:31.765017 (+ 265us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:32.016871 (+251854us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:32.749134 (+732263us) spinlock_profiling.cc:243] Waited 248 ms on lock 0x7b54001ef5d4. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787c1a1 00007fe45787abee 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc 00007fe455e9e782 00007fe455ca2bcc 00007fe455ca2006
0501 20:01:32.749302 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":570639616}
I20240501 20:01:32.689165 30762 raft_consensus.cc:504] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:32.752765 29658 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59142 (request call id 754) took 2377 ms (client timeout 2279 ms). Trace:
I20240501 20:01:32.752954 30762 leader_election.cc:288] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:32.753216 29658 rpcz_store.cc:269] 0501 20:01:30.375596 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:31.073326 (+697730us) service_pool.cc:225] Handling call
0501 20:01:32.752569 (+1679243us) spinlock_profiling.cc:243] Waited 1.44 s on lock 0x7b5400362254. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787f4fe 00007fe45c7bd9d7 00007fe455ea1c14 00007fe455ea1ba2 00007fe455ea1ad1 00007fe455ea1a4c 00007fe455ea0d02 00007fe455ca2bcc 00007fe455ca2006 00007fe455ca19af
0501 20:01:32.752701 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3302461312}
I20240501 20:01:32.759407 30848 consensus_queue.cc:1035] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:32.760125 29651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e89c0572e4ee7be8e5315b459bdb3" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:32.744084 30759 raft_consensus.cc:504] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:32.753727 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07bdb8b09b8c4ef4b8e49daa9b8c01d6" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:32.745851 30832 raft_consensus.cc:686] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [term 3 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:32.756458 29659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59142 (request call id 757) took 2380 ms. Trace:
I20240501 20:01:32.758378 29625 leader_election.cc:302] T add9d7579351483fb268f89a44e95d8a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: 888d99522e4844118f8f1cf26f268c8c, ee0d97247d394a8facfa774f15f3a67d
W20240501 20:01:32.761809 29734 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 751) took 2201 ms (client timeout 2081 ms). Trace:
I20240501 20:01:32.747422 29806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41864 (request call id 781) took 1042 ms. Trace:
I20240501 20:01:32.753787 29698 leader_election.cc:302] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:32.763008 29803 raft_consensus.cc:2454] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 3.
W20240501 20:01:32.786605 29734 rpcz_store.cc:269] 0501 20:01:30.560449 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:31.063275 (+502826us) service_pool.cc:225] Handling call
0501 20:01:32.761639 (+1698364us) spinlock_profiling.cc:243] Waited 249 ms on lock 0x7b54001af254. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787f4fe 00007fe45c7bd9d7 00007fe455ea1c14 00007fe455ea1ba2 00007fe455ea1ad1 00007fe455ea1a4c 00007fe455ea0d02 00007fe455ca2bcc 00007fe455ca2006 00007fe455ca19af
0501 20:01:32.761748 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":572098304}
I20240501 20:01:32.786787 29659 rpcz_store.cc:276] 0501 20:01:30.376457 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:31.548832 (+1172375us) service_pool.cc:225] Handling call
0501 20:01:31.549190 (+ 358us) raft_consensus.cc:1457] Updating replica for 1 ops
0501 20:01:32.564070 (+1014880us) spinlock_profiling.cc:243] Waited 3.37 ms on lock 0x7b1800081ee0. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe45d7f2661 00007fe45d7ff3e1 00007fe45750a75c 00007fe45493354c 00007fe45d76c939 00007fe45495042b 00007fe45494e82f 00007fe45494dcc5 00007fe457873620 00007fe4578741bb 00007fe45787e9db
0501 20:01:32.564223 (+ 153us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:32.564262 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0501 20:01:32.571663 (+ 7401us) log.cc:842] Serialized 26 byte log entry
0501 20:01:32.685506 (+113843us) raft_consensus.cc:1631] Marking committed up to 1
0501 20:01:32.685559 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:32.685647 (+ 88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0501 20:01:32.756144 (+ 70497us) raft_consensus.cc:1671] finished
0501 20:01:32.756171 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:32.756404 (+ 233us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31783,"raft.queue_time_us":44018,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":17879,"spinlock_wait_cycles":7751552,"thread_start_us":170028,"threads_started":2,"wal-append.queue_time_us":154471}
W20240501 20:01:32.791352 29808 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41864 (request call id 774) took 2455 ms (client timeout 1917 ms). Trace:
I20240501 20:01:32.795667 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac2171683884f6f97cdfeb0a656cb49" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:32.795804 29808 rpcz_store.cc:269] 0501 20:01:30.336388 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:30.356259 (+ 19871us) service_pool.cc:225] Handling call
0501 20:01:32.791155 (+2434896us) spinlock_profiling.cc:243] Waited 159 ms on lock 0x7b540014f5d4. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787f4fe 00007fe45c7bd9d7 00007fe455ea1c14 00007fe455ea1ba2 00007fe455ea1ad1 00007fe455ea1a4c 00007fe455ea0d02 00007fe455ca2bcc 00007fe455ca2006 00007fe455ca19af
0501 20:01:32.791250 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":366573312}
I20240501 20:01:32.797494 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0374bb790ed345ed90452595f71383a2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:32.798206 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d2cac25f354a97b1bfb53d57ae7c08" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:32.798225 29808 raft_consensus.cc:3037] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:32.798996 29803 raft_consensus.cc:3037] T d1d2cac25f354a97b1bfb53d57ae7c08 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:32.801856 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e38a2e91a3d48d09a22abce723af833" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:32.806018 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36057b3f24d44b9aac3a1378053a395f" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
W20240501 20:01:32.792737 30795 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.561s user 0.007s sys 0.005s
I20240501 20:01:32.807917 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e9f75387564275a60d38e9a910319c" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:32.810351 29808 raft_consensus.cc:2454] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 3.
W20240501 20:01:32.815948 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 61681.4us
I20240501 20:01:32.817025 29700 leader_election.cc:302] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:32.818254 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e89c0572e4ee7be8e5315b459bdb3" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:32.818797 29807 raft_consensus.cc:3037] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:32.753067 30886 raft_consensus.cc:504] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:32.790985 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6797d82dda2948d8a7445fcd33e44752" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:32.792248 30808 raft_consensus.cc:2726] T add9d7579351483fb268f89a44e95d8a P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:01:32.763079 30839 consensus_queue.cc:1035] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.041s
I20240501 20:01:32.787297 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c651044a305c4168805825a53d910899" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:32.823100 29803 raft_consensus.cc:2454] T d1d2cac25f354a97b1bfb53d57ae7c08 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 2.
I20240501 20:01:32.830894 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010ceabf3e5a4b1c9330b80dade765bb" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:32.794858 30829 consensus_queue.cc:1035] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20240501 20:01:32.790303 29625 leader_election.cc:302] T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:32.823597 29700 leader_election.cc:302] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:32.824111 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649b71f9f267437ca51c758dd9ca6987" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:32.832425 29808 raft_consensus.cc:2454] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 3.
W20240501 20:01:32.818080 30795 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.582s user 0.008s sys 0.005s
I20240501 20:01:32.803351 30898 raft_consensus.cc:2781] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:32.833199 29733 raft_consensus.cc:1226] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:32.788486 29806 rpcz_store.cc:276] 0501 20:01:31.704963 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:31.706119 (+ 1156us) service_pool.cc:225] Handling call
0501 20:01:32.747244 (+1041125us) spinlock_profiling.cc:243] Waited 967 ms on lock 0x7b540018e554. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787f4fe 00007fe45c7bd9d7 00007fe455ea1c14 00007fe455ea1ba2 00007fe455ea1ad1 00007fe455ea1a4c 00007fe455ea0d02 00007fe455ca2bcc 00007fe455ca2006 00007fe455ca19af
0501 20:01:32.747370 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2224628480}
I20240501 20:01:32.838035 30759 leader_election.cc:288] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:32.837922 30842 consensus_queue.cc:1035] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 20:01:32.838137 30844 consensus_queue.cc:1035] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 20:01:32.838658 30759 raft_consensus.cc:482] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:32.839107 30897 raft_consensus.cc:2781] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:32.837881 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a35893b91247a99912eba787d787cf" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:32.840023 30898 raft_consensus.cc:482] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:32.840282 29808 raft_consensus.cc:2379] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 3: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
W20240501 20:01:32.840189 29728 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 735) took 2373 ms (client timeout 2807 ms). Trace:
W20240501 20:01:32.840754 29728 rpcz_store.cc:269] 0501 20:01:30.466939 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:32.790653 (+2323714us) service_pool.cc:225] Handling call
0501 20:01:32.840127 (+ 49474us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:32.841296 30682 leader_election.cc:288] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:32.841866 30682 raft_consensus.cc:2781] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:01:32.842370 30682 raft_consensus.cc:686] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [term 3 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:32.842516 29802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69915dd5992641e3a24f7f45f1cd5c0c" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:32.843026 30646 raft_consensus.cc:504] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:32.843225 30682 consensus_queue.cc:237] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:32.859948 29652 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59136 (request call id 635) took 3817 ms (client timeout 1819 ms). Trace:
I20240501 20:01:32.843747 29802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3f29f0eeea4b119984634202e5d9cd" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:32.844795 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69915dd5992641e3a24f7f45f1cd5c0c" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:32.844954 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcf30cf66714efdac997862dafbe4a2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:32.848537 30800 consensus_queue.cc:1046] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:01:32.850219 30815 leader_election.cc:288] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:32.851534 30898 raft_consensus.cc:3037] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:32.840811 30844 consensus_queue.cc:1035] T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 20:01:32.852739 30836 raft_consensus.cc:504] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:32.841985 30884 raft_consensus.cc:504] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:32.853044 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:32.853132 29558 catalog_manager.cc:5484] T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 1 to 2, leader changed from fdfeac4190e045f290e003b2298d988d (127.27.217.194) to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 2 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:32.839569 30897 raft_consensus.cc:482] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:32.855432 30848 consensus_queue.cc:1035] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:32.857409 30822 leader_election.cc:288] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 4 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:32.858953 30847 ts_tablet_manager.cc:1888] T be383be783234855878cb8fd442c7b05 P 888d99522e4844118f8f1cf26f268c8c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240501 20:01:32.849700 30851 raft_consensus.cc:504] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:32.860219 30646 leader_election.cc:288] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:33.022812 30844 consensus_queue.cc:1046] T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 20:01:32.839114 30759 raft_consensus.cc:504] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:33.023928 30844 consensus_queue.cc:1035] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:32.973029 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649b71f9f267437ca51c758dd9ca6987" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:32.878767 29773 leader_election.cc:334] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 1.658s (SENT)
I20240501 20:01:33.025749 29803 raft_consensus.cc:2454] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 3.
W20240501 20:01:33.026314 29652 rpcz_store.cc:269] 0501 20:01:29.043011 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:29.045253 (+ 2242us) service_pool.cc:225] Handling call
0501 20:01:32.859775 (+3814522us) spinlock_profiling.cc:243] Waited 3.38 s on lock 0x7b540034e9d4. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787f4fe 00007fe45c7bd9d7 00007fe455ea1c14 00007fe455ea1ba2 00007fe455ea1ad1 00007fe455ea1a4c 00007fe455ea0d02 00007fe455ca2bcc 00007fe455ca2006 00007fe455ca19af
0501 20:01:32.859889 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7765241600}
I20240501 20:01:33.025166 29808 raft_consensus.cc:2454] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 3.
I20240501 20:01:32.871024 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0374bb790ed345ed90452595f71383a2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:32.944607 29625 leader_election.cc:302] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
W20240501 20:01:33.028934 29625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 84440.4us
I20240501 20:01:33.028615 29558 catalog_manager.cc:5484] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 1 to 2, leader changed from fdfeac4190e045f290e003b2298d988d (127.27.217.194) to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 2 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
W20240501 20:01:33.032302 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 208764us
I20240501 20:01:33.034025 29700 leader_election.cc:302] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:33.034718 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07bdb8b09b8c4ef4b8e49daa9b8c01d6" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
W20240501 20:01:32.973680 29772 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 116856us
I20240501 20:01:33.034943 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d2cac25f354a97b1bfb53d57ae7c08" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
W20240501 20:01:33.026135 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 147429us
I20240501 20:01:33.037022 29625 leader_election.cc:302] T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
W20240501 20:01:32.881412 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 90291.4us
I20240501 20:01:33.037973 29659 raft_consensus.cc:2354] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d for earlier term 2. Current term is 3.
W20240501 20:01:33.037241 29773 leader_election.cc:334] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 1.664s (SENT)
I20240501 20:01:32.860693 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ed27033ab1645e9ab0600ab635d9589" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
W20240501 20:01:33.101235 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 64054.8us
I20240501 20:01:33.027004 30847 log.cc:1196] T be383be783234855878cb8fd442c7b05 P 888d99522e4844118f8f1cf26f268c8c: Deleting WAL directory at /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714593602638719-28519-0/minicluster-data/ts-3-root/wals/be383be783234855878cb8fd442c7b05
I20240501 20:01:33.034685 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec073fe645614fe28e7837de3c22510b" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:33.102031 29660 raft_consensus.cc:2454] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 2.
I20240501 20:01:33.037952 29806 raft_consensus.cc:3037] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:33.087292 29800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41864 (request call id 793) took 1369 ms. Trace:
I20240501 20:01:33.088907 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9472c53f79ec423f82c396bd980674e2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:33.104605 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31fff64b453a423eb0f567b3171564fb" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:33.103580 29800 rpcz_store.cc:276] 0501 20:01:31.717456 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:31.819586 (+102130us) service_pool.cc:225] Handling call
0501 20:01:33.060920 (+1241334us) spinlock_profiling.cc:243] Waited 675 ms on lock 0x7b54001f3954. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787f4fe 00007fe45c7bd9d7 00007fe455ea1c14 00007fe455ea1ba2 00007fe455ea1ad1 00007fe455ea1a4c 00007fe455ea0d02 00007fe455ca2bcc 00007fe455ca2006 00007fe455ca19af
0501 20:01:33.087234 (+ 26314us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1552339200}
I20240501 20:01:33.105275 29653 raft_consensus.cc:2454] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 3.
I20240501 20:01:33.106201 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c597f15e844f30837012baed4b0501" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:33.087844 29558 catalog_manager.cc:5484] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 1 to 2, leader changed from fdfeac4190e045f290e003b2298d988d (127.27.217.194) to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 2 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:33.089035 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f5d1686214485d91a0baf4b870740e" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:33.088551 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e89c0572e4ee7be8e5315b459bdb3" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:33.109815 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
W20240501 20:01:33.088744 29625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 51791.1us
I20240501 20:01:32.833235 30840 leader_election.cc:288] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:32.832762 30849 consensus_queue.cc:1035] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.267s
I20240501 20:01:33.112946 30849 raft_consensus.cc:3032] T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:01:33.113289 30849 raft_consensus.cc:729] T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:33.114141 30849 consensus_queue.cc:260] T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:33.115412 30849 raft_consensus.cc:3037] T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:33.099859 29698 leader_election.cc:398] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Vote denied by peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959) with higher term. Message: Invalid argument: T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d for earlier term 4. Current term is 5.
I20240501 20:01:32.865242 29807 raft_consensus.cc:2454] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 888d99522e4844118f8f1cf26f268c8c in term 4.
I20240501 20:01:33.117627 29698 leader_election.cc:401] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20240501 20:01:33.118356 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c597f15e844f30837012baed4b0501" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:33.119081 29807 raft_consensus.cc:3037] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:33.119918 29700 leader_election.cc:302] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:33.101364 29655 raft_consensus.cc:3037] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 4
I20240501 20:01:33.037933 30646 raft_consensus.cc:2726] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:01:33.121351 29700 leader_election.cc:302] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:33.106755 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece8d4ed84c34b748f7b6272b31c90ac" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:33.110404 29653 raft_consensus.cc:2362] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 3. Re-sending same reply.
I20240501 20:01:33.124917 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5525f546426c47a38edf7f56ad04b7ab" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:33.125957 29653 raft_consensus.cc:2379] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 3: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:33.109934 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece8d4ed84c34b748f7b6272b31c90ac" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:33.126947 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add9d7579351483fb268f89a44e95d8a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:33.129689 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649b71f9f267437ca51c758dd9ca6987" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:33.130518 29653 raft_consensus.cc:2362] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 3. Re-sending same reply.
I20240501 20:01:33.135470 29803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41858 (request call id 748) took 1246 ms. Trace:
I20240501 20:01:32.860395 30808 raft_consensus.cc:2781] T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:33.097752 30906 raft_consensus.cc:2781] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:01:33.136602 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e49dd5180f04313ab33942fb3ddc147" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:33.136018 29803 rpcz_store.cc:276] 0501 20:01:31.888805 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:33.088132 (+1199327us) service_pool.cc:225] Handling call
0501 20:01:33.134971 (+ 46839us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:33.135176 (+ 205us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:33.135197 (+ 21us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:33.135231 (+ 34us) raft_consensus.cc:1631] Marking committed up to 1
0501 20:01:33.135258 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:33.135312 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:33.135405 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:33.127619 29800 raft_consensus.cc:3037] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:33.142805 29558 catalog_manager.cc:5484] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 1 to 2, leader changed from fdfeac4190e045f290e003b2298d988d (127.27.217.194) to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 2 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:33.149648 29625 leader_election.cc:302] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:33.026082 30897 raft_consensus.cc:3037] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:33.135754 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a35893b91247a99912eba787d787cf" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:33.092245 30839 raft_consensus.cc:2726] T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:33.152312 29660 raft_consensus.cc:2354] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d for earlier term 2. Current term is 3.
I20240501 20:01:33.097827 30809 leader_election.cc:288] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:33.157819 29655 raft_consensus.cc:2454] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 888d99522e4844118f8f1cf26f268c8c in term 4.
I20240501 20:01:33.167374 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8cd40cb4044d18a86001ce6ddf97ab" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:33.161020 29807 raft_consensus.cc:2454] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 4.
I20240501 20:01:33.165385 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "456035a1cb3f4f7da52d0fc1dd1c656c" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:33.087358 30909 raft_consensus.cc:482] T 010ceabf3e5a4b1c9330b80dade765bb P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:33.098395 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:33.168977 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010ceabf3e5a4b1c9330b80dade765bb" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:33.175112 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f261d2a05bf4c838532049601694b4a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:33.177405 29731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 771) took 2002 ms. Trace:
I20240501 20:01:33.185523 29731 rpcz_store.cc:276] 0501 20:01:31.176577 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:31.743129 (+566552us) service_pool.cc:225] Handling call
0501 20:01:33.177231 (+1434102us) spinlock_profiling.cc:243] Waited 709 ms on lock 0x7b5400154854. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787f4fe 00007fe45c7bd9d7 00007fe455ea1c14 00007fe455ea1ba2 00007fe455ea1ad1 00007fe455ea1a4c 00007fe455ea0d02 00007fe455ca2bcc 00007fe455ca2006 00007fe455ca19af
0501 20:01:33.177353 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1631149312}
I20240501 20:01:33.173182 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38cbeb0dc0c14e3d9182101eea885625" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:33.169195 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b337b8e3e38d4ad6b1076fbd0d0a1609" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:33.140384 30808 raft_consensus.cc:482] T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:33.134877 30759 leader_election.cc:288] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:33.161618 29804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41858 (request call id 752) took 1271 ms. Trace:
W20240501 20:01:33.134636 30904 log.cc:924] Time spent T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c: Append to log took a long time: real 0.117s user 0.003s sys 0.000s
I20240501 20:01:33.172854 29735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47414 (request call id 21) took 1986 ms. Trace:
I20240501 20:01:33.179333 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c651044a305c4168805825a53d910899" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:33.191500 29660 raft_consensus.cc:2454] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
I20240501 20:01:33.192442 29735 rpcz_store.cc:276] 0501 20:01:31.186091 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:32.272600 (+1086509us) service_pool.cc:225] Handling call
0501 20:01:32.282932 (+ 10332us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:32.473107 (+190175us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:33.172615 (+699508us) spinlock_profiling.cc:243] Waited 170 ms on lock 0x7b5400153e54. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787c1a1 00007fe45787abee 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc 00007fe455e9e782 00007fe455ca2bcc 00007fe455ca2006
0501 20:01:33.172790 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":392127232}
I20240501 20:01:33.170388 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcf30cf66714efdac997862dafbe4a2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:33.140821 30894 raft_consensus.cc:3037] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Advancing to term 5
I20240501 20:01:33.182161 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d88139f89c47ff84cad674147aae0f" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:33.168217 30886 leader_election.cc:288] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:33.195936 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8edcd903224d48f889b46e1b5318e0ec" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:33.187249 29659 raft_consensus.cc:2454] T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 2.
I20240501 20:01:33.188283 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69915dd5992641e3a24f7f45f1cd5c0c" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:33.189031 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a2bba62f434d1c8489b55c8fec2187" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:33.197105 29653 raft_consensus.cc:2379] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d in current term 3: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:33.175309 30753 raft_consensus.cc:2781] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:32.835731 30832 consensus_queue.cc:237] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:33.104159 30877 raft_consensus.cc:504] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:33.193979 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 72693.3us
I20240501 20:01:33.176087 30713 raft_consensus.cc:2781] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:33.198464 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec073fe645614fe28e7837de3c22510b" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:33.198709 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9472c53f79ec423f82c396bd980674e2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:33.217654 29807 raft_consensus.cc:2454] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
I20240501 20:01:33.218624 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d86f8848a184a87947fd927c44e5a35" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:33.199558 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5525fc4b82a44b0991c3dd6098fde81" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:33.139256 30806 raft_consensus.cc:504] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:33.183818 30852 raft_consensus.cc:3032] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:01:33.198463 29800 raft_consensus.cc:2454] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 4.
I20240501 20:01:33.215214 29700 leader_election.cc:302] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:33.176018 30827 raft_consensus.cc:2781] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:33.177812 30794 raft_consensus.cc:482] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:33.217360 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49da4246f5f6438abe0e7e8c9a523d80" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:33.200116 29655 raft_consensus.cc:2354] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d for earlier term 2. Current term is 3.
I20240501 20:01:33.219926 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb27193abbf4644a48662909bc7b04a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:33.223511 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05df43bdfa479b902b1da297fb5974" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:33.274832 29807 raft_consensus.cc:2396] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240501 20:01:33.275337 29809 raft_consensus.cc:2379] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 888d99522e4844118f8f1cf26f268c8c in current term 3: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:33.275848 29809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49474 (request call id 28) took 1625 ms. Trace:
I20240501 20:01:33.276371 29809 rpcz_store.cc:276] 0501 20:01:31.650457 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:33.223284 (+1572827us) service_pool.cc:225] Handling call
0501 20:01:33.275782 (+ 52498us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:33.277464 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48fd0a8e289401aa7417c42ff2c81b2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:33.278120 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 63024.2us
I20240501 20:01:33.194909 30808 raft_consensus.cc:3037] T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:33.178447 29654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcf30cf66714efdac997862dafbe4a2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:33.283591 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece8d4ed84c34b748f7b6272b31c90ac" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:33.180878 30909 raft_consensus.cc:504] T 010ceabf3e5a4b1c9330b80dade765bb P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:33.200305 29802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48fd0a8e289401aa7417c42ff2c81b2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:33.208252 30753 raft_consensus.cc:482] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:33.251775 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77be7101c0fe4085b6706296f609870a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:33.268311 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668b4a4d782c4278932dd8413dc3ab3e" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:33.250334 29772 leader_election.cc:398] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: Vote denied by peer 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227) with higher term. Message: Invalid argument: T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d for earlier term 2. Current term is 3.
I20240501 20:01:33.216606 30713 raft_consensus.cc:482] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:33.275686 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af08384057a4d85977481472f460168" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:33.226668 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69915dd5992641e3a24f7f45f1cd5c0c" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:33.290163 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77be7101c0fe4085b6706296f609870a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:33.290598 29660 raft_consensus.cc:2454] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 2.
I20240501 20:01:33.278728 29700 leader_election.cc:302] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:33.291427 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6797d82dda2948d8a7445fcd33e44752" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:33.278226 29809 raft_consensus.cc:2454] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 4.
I20240501 20:01:33.187815 30383 leader_election.cc:302] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, 888d99522e4844118f8f1cf26f268c8c; no voters:
W20240501 20:01:33.222995 30384 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 198627us
I20240501 20:01:33.225404 30761 leader_election.cc:288] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:33.273942 30827 raft_consensus.cc:482] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:33.294488 29700 leader_election.cc:302] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:33.296656 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0374bb790ed345ed90452595f71383a2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:33.297641 29809 raft_consensus.cc:2379] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 3: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:33.285363 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a35893b91247a99912eba787d787cf" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:33.190460 30847 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45668 (request call id 0) took 3198 ms. Trace:
I20240501 20:01:33.283861 30886 raft_consensus.cc:2781] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:01:33.250046 30852 raft_consensus.cc:729] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:33.289481 29772 leader_election.cc:401] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
W20240501 20:01:33.292254 29773 leader_election.cc:334] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 1.818s (SENT)
I20240501 20:01:33.293649 29660 raft_consensus.cc:2354] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d for earlier term 2. Current term is 3.
I20240501 20:01:33.295583 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e38a2e91a3d48d09a22abce723af833" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:33.285113 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668b4a4d782c4278932dd8413dc3ab3e" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:33.306208 29653 raft_consensus.cc:3037] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:33.307523 29654 raft_consensus.cc:2379] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 3: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:33.308372 29654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c597f15e844f30837012baed4b0501" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:33.289961 30753 raft_consensus.cc:3037] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:33.309329 29800 raft_consensus.cc:1261] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20240501 20:01:33.309478 29654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07bdb8b09b8c4ef4b8e49daa9b8c01d6" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:33.310415 30418 raft_consensus.cc:1226] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [term 4 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 2. Current term is 4. Ops: []
I20240501 20:01:33.310603 29654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af08384057a4d85977481472f460168" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:33.311218 29654 raft_consensus.cc:3037] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:33.311499 30417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "696121417ea9403990456cfc6fc4f416" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c"
I20240501 20:01:33.305404 30836 leader_election.cc:288] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:33.312714 30414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05df43bdfa479b902b1da297fb5974" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c"
I20240501 20:01:33.313153 30877 leader_election.cc:288] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:33.313503 30414 raft_consensus.cc:2379] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d in current term 3: Already voted for candidate 888d99522e4844118f8f1cf26f268c8c in this term.
I20240501 20:01:33.305552 30908 raft_consensus.cc:2781] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:01:33.313932 30884 leader_election.cc:288] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 5 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:33.314635 30385 leader_election.cc:302] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 888d99522e4844118f8f1cf26f268c8c, ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f
W20240501 20:01:33.314824 30383 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 133868us
I20240501 20:01:33.316062 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8cd40cb4044d18a86001ce6ddf97ab" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:33.300931 30827 raft_consensus.cc:3037] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:33.317190 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca3948261374cef8b1b206a43c00173" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:33.283604 30794 raft_consensus.cc:504] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:33.318181 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458740ab5c444b6ab3d1991c69db5084" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:33.314841 30886 raft_consensus.cc:686] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [term 3 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:33.319073 29657 raft_consensus.cc:2379] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d in current term 3: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:33.319761 30886 consensus_queue.cc:237] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:33.293244 30713 raft_consensus.cc:3037] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Advancing to term 4
W20240501 20:01:33.323294 29772 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.196:37227, user_credentials={real_user=slave}} blocked reactor thread for 73321.1us
I20240501 20:01:33.308497 30851 leader_election.cc:288] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:33.309729 29800 pending_rounds.cc:76] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d: Aborting all ops after (but not including) 1
I20240501 20:01:33.303089 30847 rpcz_store.cc:276] 0501 20:01:30.204034 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:30.243842 (+ 39808us) service_pool.cc:225] Handling call
0501 20:01:30.682735 (+438893us) ts_tablet_manager.cc:1120] Deleting tablet be383be783234855878cb8fd442c7b05
0501 20:01:30.754556 (+ 71821us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0501 20:01:32.751209 (+1996653us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:32.858717 (+107508us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:33.182564 (+323847us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1736260,"tablet-delete.queue_time_us":185287,"thread_start_us":87183,"threads_started":1}
I20240501 20:01:33.302080 29655 raft_consensus.cc:2379] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d in current term 3: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:33.306356 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece8d4ed84c34b748f7b6272b31c90ac" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:33.303959 29809 raft_consensus.cc:1261] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240501 20:01:33.195070 29804 rpcz_store.cc:276] 0501 20:01:31.889793 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:32.079057 (+189264us) service_pool.cc:225] Handling call
0501 20:01:32.844658 (+765601us) spinlock_profiling.cc:243] Waited 15.6 ms on lock 0x7b58000d2c98. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe454797b51 00007fe4549193b1 00007fe454919262 00007fe45d80b889 00007fe45c7483c0 00007fe45c80e3bb 00007fe45c80e441 00007fe45c7bdf60 00007fe45c7bd844 00007fe455ea1c14 00007fe455ea1ba2
0501 20:01:33.150690 (+306032us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":35869440}
I20240501 20:01:33.301415 29807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41858 (request call id 756) took 1390 ms. Trace:
W20240501 20:01:33.324633 29773 leader_election.cc:334] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 1.819s (SENT)
I20240501 20:01:33.191804 29805 raft_consensus.cc:2454] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 3.
I20240501 20:01:33.320001 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77be7101c0fe4085b6706296f609870a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:33.329056 29800 pending_rounds.cc:97] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20240501 20:01:33.347091 29800 raft_consensus.cc:2866] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20240501 20:01:33.349187 29773 leader_election.cc:334] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 1.872s (SENT)
W20240501 20:01:33.350014 29773 leader_election.cc:334] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 1.876s (SENT)
I20240501 20:01:33.350435 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "696121417ea9403990456cfc6fc4f416" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:33.351327 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e38a2e91a3d48d09a22abce723af833" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:33.347445 29653 raft_consensus.cc:2454] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 3.
I20240501 20:01:33.455596 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05df43bdfa479b902b1da297fb5974" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
W20240501 20:01:33.425657 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 131447us
I20240501 20:01:33.368669 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd5e83faaff4e2da4e4cc22ba410d33" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:33.408932 29801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36057b3f24d44b9aac3a1378053a395f" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:33.486097 29700 leader_election.cc:302] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:33.347509 29807 rpcz_store.cc:276] 0501 20:01:31.911515 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:33.300498 (+1388983us) service_pool.cc:225] Handling call
0501 20:01:33.300833 (+ 335us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:33.301029 (+ 196us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:33.301074 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:33.301128 (+ 54us) raft_consensus.cc:1631] Marking committed up to 1
0501 20:01:33.301170 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:33.301236 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:33.301353 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:33.405961 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ed27033ab1645e9ab0600ab635d9589" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:33.455929 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9046e38fc57f4364aa3d539e1fc01928" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
W20240501 20:01:33.486991 29801 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41858 (request call id 755) took 1596 ms (client timeout 1960 ms). Trace:
I20240501 20:01:33.484620 29657 raft_consensus.cc:2379] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d in current term 2: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
W20240501 20:01:33.488579 29801 rpcz_store.cc:269] 0501 20:01:31.890581 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:32.239663 (+349082us) service_pool.cc:225] Handling call
0501 20:01:33.408776 (+1169113us) spinlock_profiling.cc:243] Waited 5.21 ms on lock 0x7b58000d2c98. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe454797b51 00007fe4549193b1 00007fe454919262 00007fe45d80b889 00007fe45c7483c0 00007fe45c80e3bb 00007fe45c80e441 00007fe45c7bdf60 00007fe45c7bd844 00007fe455ea1c14 00007fe455ea1ba2
0501 20:01:33.486934 (+ 78158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11981056}
I20240501 20:01:33.347961 29800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41858 (request call id 763) took 1065 ms. Trace:
I20240501 20:01:33.489557 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649b71f9f267437ca51c758dd9ca6987" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:33.352247 29809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41858 (request call id 762) took 1081 ms. Trace:
I20240501 20:01:33.486065 29660 raft_consensus.cc:2454] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 2.
W20240501 20:01:33.485060 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 135073us
I20240501 20:01:33.487974 29734 raft_consensus.cc:3037] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Advancing to term 4
W20240501 20:01:33.492734 29773 leader_election.cc:334] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 2.008s (SENT)
I20240501 20:01:33.491842 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "456035a1cb3f4f7da52d0fc1dd1c656c" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:33.493809 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d2cac25f354a97b1bfb53d57ae7c08" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:33.489825 29800 rpcz_store.cc:276] 0501 20:01:32.283046 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:33.308752 (+1025706us) service_pool.cc:225] Handling call
0501 20:01:33.309010 (+ 258us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:33.347744 (+ 38734us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:33.347913 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:33.494596 29732 raft_consensus.cc:1261] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240501 20:01:33.494848 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8787bc4d6b44649b9138d906730110a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:33.395769 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d88139f89c47ff84cad674147aae0f" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:33.491210 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211e27a3b0084b89a6dd7550849b5a65" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:33.496420 29730 raft_consensus.cc:3037] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:01:33.497105 29773 leader_election.cc:334] T 3c5345f6eb62449ba69b4a9c2e349b31 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 1.534s (SENT)
I20240501 20:01:33.498437 29625 leader_election.cc:302] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:33.499421 29734 raft_consensus.cc:2454] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 888d99522e4844118f8f1cf26f268c8c in term 4.
I20240501 20:01:33.499820 29625 leader_election.cc:302] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:33.500779 29625 leader_election.cc:302] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:33.489753 29700 leader_election.cc:302] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:33.498900 29801 raft_consensus.cc:1261] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240501 20:01:33.488591 29731 raft_consensus.cc:2454] T 9046e38fc57f4364aa3d539e1fc01928 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 888d99522e4844118f8f1cf26f268c8c in term 1.
I20240501 20:01:33.502104 29730 raft_consensus.cc:2454] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 2.
I20240501 20:01:33.491590 29809 rpcz_store.cc:276] 0501 20:01:32.271209 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:33.303102 (+1031893us) service_pool.cc:225] Handling call
0501 20:01:33.303436 (+ 334us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:33.351910 (+ 48474us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:33.352184 (+ 274us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:33.492327 30759 raft_consensus.cc:482] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:33.433424 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd5e83faaff4e2da4e4cc22ba410d33" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:33.503782 30759 raft_consensus.cc:504] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:33.504868 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd5e83faaff4e2da4e4cc22ba410d33" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:33.505441 29653 raft_consensus.cc:3037] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:33.505739 30759 leader_election.cc:288] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:33.506073 29733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 767) took 2341 ms. Trace:
I20240501 20:01:33.506259 30650 raft_consensus.cc:2726] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:33.506878 30646 raft_consensus.cc:2726] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:01:33.492893 30415 raft_consensus.cc:1203] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [term 3 LEADER]: Deduplicated request from leader. Original: 1.4->[2.5-2.5] Dedup: 2.5->[]
I20240501 20:01:33.508185 30415 raft_consensus.cc:1226] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [term 3 LEADER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20240501 20:01:33.510382 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0374bb790ed345ed90452595f71383a2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:33.511139 29659 raft_consensus.cc:2379] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d in current term 3: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
W20240501 20:01:33.492265 29698 connection.cc:232] RPC call timeout handler was delayed by -1.27157234099991s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1787s for a response
I20240501 20:01:33.511723 30894 raft_consensus.cc:2726] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959) with higher term. Message: Invalid argument: T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d for earlier term 4. Current term is 5.
I20240501 20:01:33.512495 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49da4246f5f6438abe0e7e8c9a523d80" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:33.455011 30753 raft_consensus.cc:504] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:33.513772 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f261d2a05bf4c838532049601694b4a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:33.514345 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6797d82dda2948d8a7445fcd33e44752" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:33.328516 30822 raft_consensus.cc:2781] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:01:33.515089 29656 raft_consensus.cc:2379] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 3: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:33.511886 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d86f8848a184a87947fd927c44e5a35" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:33.515504 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0374bb790ed345ed90452595f71383a2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:33.328661 30912 raft_consensus.cc:2726] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227) with higher term. Message: Invalid argument: T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d for earlier term 2. Current term is 3.
I20240501 20:01:33.515965 29659 raft_consensus.cc:2454] T 2d86f8848a184a87947fd927c44e5a35 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 2.
I20240501 20:01:33.515923 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b337b8e3e38d4ad6b1076fbd0d0a1609" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:33.519191 29656 raft_consensus.cc:2454] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 2.
I20240501 20:01:33.519421 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:33.418303 29727 raft_consensus.cc:1226] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [term 3 LEADER]: Rejecting Update request from peer 888d99522e4844118f8f1cf26f268c8c for earlier term 2. Current term is 3. Ops: []
I20240501 20:01:33.405557 29802 raft_consensus.cc:1226] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 2. Current term is 3. Ops: []
I20240501 20:01:33.520392 29660 raft_consensus.cc:2379] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d in current term 3: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:33.428123 30682 raft_consensus.cc:2781] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:33.517444 29625 leader_election.cc:302] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:33.449684 30806 leader_election.cc:288] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:33.435196 29806 raft_consensus.cc:2454] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 3.
I20240501 20:01:33.517910 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:33.329399 30884 raft_consensus.cc:2781] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:01:33.519737 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
W20240501 20:01:33.408890 30385 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 45551us
I20240501 20:01:33.419755 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f5d1686214485d91a0baf4b870740e" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:33.426765 30761 raft_consensus.cc:482] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:33.516857 29800 raft_consensus.cc:1261] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240501 20:01:33.493968 30921 raft_consensus.cc:2781] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:01:33.521459 30829 raft_consensus.cc:2781] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:33.335567 30908 raft_consensus.cc:686] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:33.522827 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458740ab5c444b6ab3d1991c69db5084" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:33.536245 29735 raft_consensus.cc:3037] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:33.531494 29653 raft_consensus.cc:2454] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 888d99522e4844118f8f1cf26f268c8c in term 5.
I20240501 20:01:33.506446 29733 rpcz_store.cc:276] 0501 20:01:31.164189 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:32.251974 (+1087785us) service_pool.cc:225] Handling call
0501 20:01:32.363439 (+111465us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:32.836853 (+473414us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:33.505074 (+668221us) spinlock_profiling.cc:243] Waited 470 ms on lock 0x7b5400371254. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787c1a1 00007fe45787abee 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc 00007fe455e9e782 00007fe455ca2bcc 00007fe455ca2006
0501 20:01:33.505906 (+ 832us) spinlock_profiling.cc:243] Waited 111 ms on lock 0x7b5400371250. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787ac59 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc 00007fe455e9e782 00007fe455ca2bcc 00007fe455ca2006 00007fe455ca19af
0501 20:01:33.506023 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1335617024}
I20240501 20:01:33.328367 30906 raft_consensus.cc:686] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [term 2 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:33.414917 29542 catalog_manager.cc:4830] TS 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227): tablet be383be783234855878cb8fd442c7b05 (table test-workload [id=c4cd6efa2a67402dac287d449c66135b]) successfully deleted
W20240501 20:01:33.538095 29542 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.27.217.196:37227, user_credentials={real_user=slave}} blocked reactor thread for 132252us
I20240501 20:01:33.530752 29802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41858 (request call id 765) took 1222 ms. Trace:
I20240501 20:01:33.436539 30794 leader_election.cc:288] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:33.449227 29805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49474 (request call id 30) took 1032 ms. Trace:
I20240501 20:01:33.395689 30917 raft_consensus.cc:482] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:33.435573 30649 leader_election.cc:288] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:33.535773 30829 raft_consensus.cc:482] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:33.523799 29625 leader_election.cc:302] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:33.446107 29651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8787bc4d6b44649b9138d906730110a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:33.549391 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36057b3f24d44b9aac3a1378053a395f" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:33.528288 29808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41858 (request call id 753) took 1638 ms. Trace:
I20240501 20:01:33.436676 30909 leader_election.cc:288] T 010ceabf3e5a4b1c9330b80dade765bb P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:33.550592 29808 rpcz_store.cc:276] 0501 20:01:31.890031 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:33.054958 (+1164927us) service_pool.cc:225] Handling call
0501 20:01:33.507888 (+452930us) spinlock_profiling.cc:243] Waited 11 ms on lock 0x7b58000d2c98. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe454797b51 00007fe4549193b1 00007fe454919262 00007fe45d80b889 00007fe45c7483c0 00007fe45c80e3bb 00007fe45c80e441 00007fe45c7d2060 00007fe45c7bd5a9 00007fe45c7bd052 00007fe455e9f694
0501 20:01:33.527585 (+ 19697us) spinlock_profiling.cc:243] Waited 10.4 ms on lock 0x7b58000d2c98. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe454797b51 00007fe4549196dc 00007fe45491929b 00007fe45d80b889 00007fe45c7483c0 00007fe45c80e3bb 00007fe45c80e441 00007fe45c7d2060 00007fe45c7bd5a9 00007fe45c7bd052 00007fe455e9f694
0501 20:01:33.527739 (+ 154us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:33.527884 (+ 145us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:33.527933 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:33.528035 (+ 102us) raft_consensus.cc:1631] Marking committed up to 2
0501 20:01:33.528090 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:33.528143 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:33.528240 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":49164928}
I20240501 20:01:33.551518 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649b71f9f267437ca51c758dd9ca6987" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:33.529892 29656 raft_consensus.cc:2454] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 3.
W20240501 20:01:33.511689 29698 connection.cc:232] RPC call timeout handler was delayed by -1.1851723409999977s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18830000000000002s for a response
I20240501 20:01:33.552132 29808 raft_consensus.cc:3037] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:33.550148 29732 raft_consensus.cc:2379] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 2: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:33.551450 29625 leader_election.cc:302] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:33.554126 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f5d1686214485d91a0baf4b870740e" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:33.545528 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c651044a305c4168805825a53d910899" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:33.555325 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31fff64b453a423eb0f567b3171564fb" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:33.521260 30851 raft_consensus.cc:2726] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:01:33.326328 30836 raft_consensus.cc:482] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:33.514503 30753 leader_election.cc:288] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:33.531618 29772 leader_election.cc:302] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:33.529541 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd5e83faaff4e2da4e4cc22ba410d33" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:33.535465 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfac893b4b4f457694a4eb184b6abb53" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:33.523926 30822 raft_consensus.cc:686] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [term 4 LEADER]: Becoming Leader. State: Replica: 888d99522e4844118f8f1cf26f268c8c, State: Running, Role: LEADER
I20240501 20:01:33.525442 30682 raft_consensus.cc:482] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:33.526795 30769 raft_consensus.cc:2726] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:01:33.550107 30829 raft_consensus.cc:3037] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:33.540493 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e9f75387564275a60d38e9a910319c" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:33.540985 29802 rpcz_store.cc:276] 0501 20:01:32.308707 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:33.313047 (+1004340us) service_pool.cc:225] Handling call
0501 20:01:33.335305 (+ 22258us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:33.525398 (+190093us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:33.530690 (+ 5292us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:33.529282 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c597f15e844f30837012baed4b0501" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:33.520535 30759 consensus_queue.cc:1035] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20240501 20:01:33.543733 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1a08b455b94ae089396a15c837ddc2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:33.590971 29731 raft_consensus.cc:3037] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:33.515453 30827 raft_consensus.cc:504] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:33.352294 30852 consensus_queue.cc:260] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:33.526290 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48630ae6eb394ac1b54955dd01cf0ed2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:33.535300 30884 raft_consensus.cc:686] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 4 LEADER]: Becoming Leader. State: Replica: 888d99522e4844118f8f1cf26f268c8c, State: Running, Role: LEADER
I20240501 20:01:33.555644 29733 raft_consensus.cc:2454] T c651044a305c4168805825a53d910899 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:33.597188 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5345f6eb62449ba69b4a9c2e349b31" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:33.558036 29625 leader_election.cc:302] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:33.558352 29735 raft_consensus.cc:2454] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 3.
I20240501 20:01:33.537726 30761 raft_consensus.cc:504] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:33.541512 30921 raft_consensus.cc:686] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:33.530158 30898 raft_consensus.cc:504] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:33.567621 29772 leader_election.cc:398] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: Vote denied by peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959) with higher term. Message: Invalid argument: T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d for earlier term 2. Current term is 3.
I20240501 20:01:33.572209 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d88139f89c47ff84cad674147aae0f" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:33.446516 30897 raft_consensus.cc:504] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:33.577953 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36057b3f24d44b9aac3a1378053a395f" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:33.549166 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a2bba62f434d1c8489b55c8fec2187" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:33.579665 29734 raft_consensus.cc:3037] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:33.497431 29773 leader_election.cc:302] T 3c5345f6eb62449ba69b4a9c2e349b31 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:33.607558 29660 raft_consensus.cc:1226] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20240501 20:01:33.580852 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d2cac25f354a97b1bfb53d57ae7c08" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:33.571949 30912 raft_consensus.cc:2726] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 20:01:33.582176 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0374bb790ed345ed90452595f71383a2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:33.581853 30908 consensus_queue.cc:237] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:33.583436 29809 raft_consensus.cc:1226] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20240501 20:01:33.565582 30649 raft_consensus.cc:2726] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:33.544862 30383 leader_election.cc:302] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, 888d99522e4844118f8f1cf26f268c8c; no voters:
I20240501 20:01:33.548553 30917 raft_consensus.cc:504] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:33.574571 30682 raft_consensus.cc:3037] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:33.598593 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9472c53f79ec423f82c396bd980674e2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:33.599572 29772 leader_election.cc:401] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240501 20:01:33.599373 29808 raft_consensus.cc:2454] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 4.
I20240501 20:01:33.601603 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcf30cf66714efdac997862dafbe4a2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:33.566715 30808 raft_consensus.cc:504] T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:33.615329 29731 raft_consensus.cc:2454] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 2.
I20240501 20:01:33.609501 30921 consensus_queue.cc:237] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:33.570973 30836 raft_consensus.cc:504] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:33.610898 30827 leader_election.cc:288] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:33.567226 30906 consensus_queue.cc:237] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:33.617111 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 120014us
I20240501 20:01:33.618541 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48fd0a8e289401aa7417c42ff2c81b2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:33.620246 29803 raft_consensus.cc:1261] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240501 20:01:33.612519 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac2171683884f6f97cdfeb0a656cb49" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:33.645759 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84fa28fb55f44b61ba1a88ed775a78ba" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:33.669476 29652 raft_consensus.cc:2379] T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 4: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
W20240501 20:01:33.667397 29773 leader_election.cc:334] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 1.710s (SENT)
W20240501 20:01:33.671494 29773 leader_election.cc:334] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 2.039s (SENT)
W20240501 20:01:33.672734 29773 leader_election.cc:334] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 2.087s (SENT)
I20240501 20:01:33.617424 30897 leader_election.cc:288] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:33.677426 30808 leader_election.cc:288] T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:33.648648 30383 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 105507us
I20240501 20:01:33.668813 29803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41858 (request call id 759) took 1620 ms. Trace:
I20240501 20:01:33.680680 29803 rpcz_store.cc:276] 0501 20:01:32.048528 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:33.302361 (+1253833us) service_pool.cc:225] Handling call
0501 20:01:33.605566 (+303205us) spinlock_profiling.cc:243] Waited 991 us on lock 0x7b58000d2c98. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe454797b51 00007fe4549193b1 00007fe454919262 00007fe45d80b889 00007fe45c7483c0 00007fe45c80e3bb 00007fe45c80e441 00007fe45c7d2060 00007fe45c7bd5a9 00007fe45c7bd052 00007fe455e9f694
0501 20:01:33.619707 (+ 14141us) spinlock_profiling.cc:243] Waited 5.87 ms on lock 0x7b58000d2c98. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe454797b51 00007fe4549196dc 00007fe45491929b 00007fe45d80b889 00007fe45c7483c0 00007fe45c80e3bb 00007fe45c80e441 00007fe45c7d2060 00007fe45c7bd5a9 00007fe45c7bd052 00007fe455e9f694
0501 20:01:33.619849 (+ 142us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:33.668551 (+ 48702us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:33.668762 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15772416}
I20240501 20:01:33.676016 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07bdb8b09b8c4ef4b8e49daa9b8c01d6" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:33.682572 29809 raft_consensus.cc:3037] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:33.683599 29654 raft_consensus.cc:2454] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 3.
I20240501 20:01:33.685115 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211e27a3b0084b89a6dd7550849b5a65" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:33.685859 30646 raft_consensus.cc:2726] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:33.686183 29624 leader_election.cc:302] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:33.687417 30808 consensus_queue.cc:1035] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:33.678902 30872 raft_consensus.cc:2781] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:01:33.644443 30912 raft_consensus.cc:2726] T 3c5345f6eb62449ba69b4a9c2e349b31 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:01:33.691605 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458740ab5c444b6ab3d1991c69db5084" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:33.605083 30851 consensus_queue.cc:1035] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.060s
I20240501 20:01:33.699580 30808 consensus_queue.cc:1046] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 20:01:33.648577 29727 raft_consensus.cc:1226] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 3. Current term is 4. Ops: []
I20240501 20:01:33.695537 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c597f15e844f30837012baed4b0501" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:33.702646 29726 raft_consensus.cc:2362] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 2. Re-sending same reply.
I20240501 20:01:33.704708 29803 raft_consensus.cc:1226] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20240501 20:01:33.647471 29807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41864 (request call id 808) took 1079 ms. Trace:
W20240501 20:01:33.708274 29773 leader_election.cc:334] T 84fa28fb55f44b61ba1a88ed775a78ba P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 1.706s (SENT)
I20240501 20:01:33.709744 29773 leader_election.cc:302] T 84fa28fb55f44b61ba1a88ed775a78ba P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:33.543745 29805 rpcz_store.cc:276] 0501 20:01:32.417299 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:33.355499 (+938200us) service_pool.cc:225] Handling call
0501 20:01:33.398184 (+ 42685us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:33.417710 (+ 19526us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:33.419028 (+ 1318us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:33.427344 (+ 8316us) raft_consensus.cc:1631] Marking committed up to 5
0501 20:01:33.429685 (+ 2341us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:33.449018 (+ 19333us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:33.449162 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:33.651605 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:33.714362 29808 raft_consensus.cc:3037] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:33.714970 29659 raft_consensus.cc:1226] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20240501 20:01:33.716243 29659 raft_consensus.cc:1226] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20240501 20:01:33.717536 29659 raft_consensus.cc:1226] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 1. Current term is 2. Ops: []
W20240501 20:01:33.719142 29773 leader_election.cc:334] T 1ca3948261374cef8b1b206a43c00173 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 1.776s (SENT)
I20240501 20:01:33.659215 29700 leader_election.cc:302] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:33.657784 29730 raft_consensus.cc:3037] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:33.675808 29658 raft_consensus.cc:1226] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 4. Current term is 5. Ops: []
I20240501 20:01:33.689260 30831 raft_consensus.cc:2781] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election won for term 3
W20240501 20:01:33.689855 29698 service_pool.cc:135] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36260 dropped due to backpressure. The service queue is full; it has 50 items.
I20240501 20:01:33.604887 30769 consensus_queue.cc:1035] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.085s
I20240501 20:01:33.616503 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b337b8e3e38d4ad6b1076fbd0d0a1609" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:33.616113 30884 consensus_queue.cc:237] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:33.708070 29807 rpcz_store.cc:276] 0501 20:01:32.567775 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:33.487634 (+919859us) service_pool.cc:225] Handling call
0501 20:01:33.487915 (+ 281us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:33.490214 (+ 2299us) raft_consensus.cc:1520] Early marking committed up to index 2
0501 20:01:33.558604 (+ 68390us) log.cc:842] Serialized 14 byte log entry
0501 20:01:33.647150 (+ 88546us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:33.647200 (+ 50us) raft_consensus.cc:1631] Marking committed up to 2
0501 20:01:33.647243 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:33.647308 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:33.647421 (+ 113us) inbound_call.cc:173] Queueing success response
0501 20:01:33.694126 (+ 46705us) spinlock_profiling.cc:243] Waited 24.3 ms on lock 0x7b18002a83e0. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe45d7f2661 00007fe45d7ff3e1 00007fe4549511b8 00007fe454954691 00007fe454954649 00007fe4549545d9 00007fe4549545a1 00007fe45495389d 00007fe45dbc5da4 00007fe45dbc5bd9 00007fe454933936
Metrics: {"mutex_wait_us":10437,"prepare.queue_time_us":111717,"prepare.run_cpu_time_us":26,"prepare.run_wall_time_us":28,"raft.queue_time_us":10591,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":15483,"spinlock_wait_cycles":55959808,"thread_start_us":111714,"threads_started":2,"wal-append.queue_time_us":105827}
I20240501 20:01:33.657655 30849 raft_consensus.cc:2726] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959) with higher term. Message: Invalid argument: T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d for earlier term 2. Current term is 3.
I20240501 20:01:33.700179 29731 raft_consensus.cc:2362] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 3. Re-sending same reply.
I20240501 20:01:33.724642 29809 raft_consensus.cc:2454] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 4.
I20240501 20:01:33.648783 30822 consensus_queue.cc:237] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:33.623097 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e9f75387564275a60d38e9a910319c" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:33.633402 29734 raft_consensus.cc:2454] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 2.
I20240501 20:01:33.719534 29773 leader_election.cc:302] T 1ca3948261374cef8b1b206a43c00173 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:33.698390 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77be7101c0fe4085b6706296f609870a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:33.714078 30809 raft_consensus.cc:2726] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:01:33.650173 29772 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 82681.7us
I20240501 20:01:33.611419 29802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010ceabf3e5a4b1c9330b80dade765bb" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:33.649561 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49da4246f5f6438abe0e7e8c9a523d80" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:33.762785 30898 leader_election.cc:288] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:33.739750 29808 raft_consensus.cc:2454] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 4.
I20240501 20:01:33.745013 29732 raft_consensus.cc:2454] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 5.
W20240501 20:01:33.740283 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 81179.1us
I20240501 20:01:33.761531 29656 raft_consensus.cc:1226] T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:33.762457 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6797d82dda2948d8a7445fcd33e44752" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:33.690608 30649 consensus_queue.cc:1046] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.079s
I20240501 20:01:33.766052 29730 raft_consensus.cc:2454] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 2.
I20240501 20:01:33.690140 30917 leader_election.cc:288] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:33.729555 29803 raft_consensus.cc:1226] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20240501 20:01:33.802933 29700 leader_election.cc:302] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:33.803900 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e49dd5180f04313ab33942fb3ddc147" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:33.804533 29803 raft_consensus.cc:2454] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 3.
I20240501 20:01:33.805516 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48630ae6eb394ac1b54955dd01cf0ed2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:33.806632 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3f29f0eeea4b119984634202e5d9cd" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:33.697716 30761 leader_election.cc:288] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:33.767139 29655 raft_consensus.cc:1226] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:33.767938 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07bdb8b09b8c4ef4b8e49daa9b8c01d6" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:33.775995 30842 raft_consensus.cc:2726] T 1ca3948261374cef8b1b206a43c00173 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:33.697669 30682 raft_consensus.cc:504] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:33.768992 29809 raft_consensus.cc:1226] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 2. Current term is 3. Ops: []
I20240501 20:01:33.722138 30912 raft_consensus.cc:2726] T 84fa28fb55f44b61ba1a88ed775a78ba P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 20:01:33.771173 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 52060.9us
I20240501 20:01:33.645905 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be383be783234855878cb8fd442c7b05" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:33.814168 30713 raft_consensus.cc:504] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:33.815471 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48fd0a8e289401aa7417c42ff2c81b2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
W20240501 20:01:33.773773 30383 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 94780.8us
I20240501 20:01:33.759868 30831 raft_consensus.cc:482] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:33.776257 29653 raft_consensus.cc:1226] T 010ceabf3e5a4b1c9330b80dade765bb P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:33.778239 29772 leader_election.cc:302] T ec073fe645614fe28e7837de3c22510b P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:33.701056 30872 raft_consensus.cc:686] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [term 5 LEADER]: Becoming Leader. State: Replica: 888d99522e4844118f8f1cf26f268c8c, State: Running, Role: LEADER
I20240501 20:01:33.760206 30897 raft_consensus.cc:2781] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:01:33.781087 29800 raft_consensus.cc:1261] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20240501 20:01:33.781533 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 91754.4us
I20240501 20:01:33.789857 29808 raft_consensus.cc:1261] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240501 20:01:33.765903 29807 raft_consensus.cc:1226] T d1d2cac25f354a97b1bfb53d57ae7c08 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:33.691085 30839 consensus_queue.cc:1035] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.100s
I20240501 20:01:33.781818 29729 raft_consensus.cc:3037] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:33.810894 29700 leader_election.cc:302] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:33.763921 29698 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36260 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 728)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 733)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 691)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 703)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 704)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 729)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 705)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 795)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 731)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 801)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 706)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 735)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 707)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 796)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 791)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 722)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 702)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 794)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 690)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 789)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 802)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 732)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 718)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 730)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 697)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 698)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 708)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 709)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 710)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 711)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 712)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 713)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 714)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 715)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 716)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 719)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 720)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 793)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 721)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 723)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 724)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 725)
I20240501 20:01:33.772814 29801 raft_consensus.cc:1261] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:01:33.816622 29657 raft_consensus.cc:2379] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d in current term 4: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:33.800911 30836 leader_election.cc:288] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:33.823269 29803 raft_consensus.cc:1226] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Rejecting Update request from peer 888d99522e4844118f8f1cf26f268c8c for earlier term 2. Current term is 3. Ops: []
W20240501 20:01:33.823611 29772 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 45467us
I20240501 20:01:33.832976 29772 leader_election.cc:302] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:33.816231 29728 raft_consensus.cc:2379] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 3: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:33.818931 29733 raft_consensus.cc:2379] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 5: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:33.825917 29658 raft_consensus.cc:1226] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:33.826643 30713 leader_election.cc:288] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:33.826745 29800 consensus_queue.cc:237] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:33.820443 30917 raft_consensus.cc:2781] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:01:33.835247 29700 leader_election.cc:302] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:33.844466 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec073fe645614fe28e7837de3c22510b" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:33.828146 30912 raft_consensus.cc:2781] T ec073fe645614fe28e7837de3c22510b P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:33.847570 29772 leader_election.cc:302] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, 888d99522e4844118f8f1cf26f268c8c
I20240501 20:01:33.842664 29698 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36260 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 737)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 738)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 739)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 740)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 705)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 795)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 731)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 801)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 706)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 741)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 735)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 707)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 796)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 744)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 791)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 722)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 702)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 794)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 690)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 789)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 802)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 732)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 736)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 718)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 746)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 730)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 697)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 698)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 708)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 709)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 710)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 711)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 712)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 713)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 714)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 715)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 716)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 719)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 720)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 793)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 721)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 723)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 724)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 725)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 742)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 743)
I20240501 20:01:33.821626 30831 raft_consensus.cc:3037] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:33.822003 30935 raft_consensus.cc:3032] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:01:33.854691 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31fff64b453a423eb0f567b3171564fb" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:33.855639 29772 leader_election.cc:302] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters: fdfeac4190e045f290e003b2298d988d
W20240501 20:01:33.859845 29773 leader_election.cc:334] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 1.868s (SENT)
I20240501 20:01:33.863888 29773 leader_election.cc:302] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:33.864086 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d2cac25f354a97b1bfb53d57ae7c08" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:33.867197 29772 leader_election.cc:302] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:33.829094 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5345f6eb62449ba69b4a9c2e349b31" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:33.840739 30827 raft_consensus.cc:2781] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:01:33.840413 30842 raft_consensus.cc:482] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:33.868117 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9472c53f79ec423f82c396bd980674e2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:33.870493 30933 consensus_queue.cc:1035] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240501 20:01:33.846293 30906 raft_consensus.cc:482] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:33.873795 29772 leader_election.cc:302] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, 888d99522e4844118f8f1cf26f268c8c
I20240501 20:01:33.874537 29698 leader_election.cc:302] T 9046e38fc57f4364aa3d539e1fc01928 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 888d99522e4844118f8f1cf26f268c8c, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:33.859288 30849 raft_consensus.cc:2781] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:33.887473 29732 raft_consensus.cc:2379] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 2: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
W20240501 20:01:33.848026 30384 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 69977.5us
I20240501 20:01:33.869232 30866 raft_consensus.cc:2726] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:33.865705 30759 raft_consensus.cc:2781] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:33.858206 30912 raft_consensus.cc:482] T ec073fe645614fe28e7837de3c22510b P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:33.861960 30935 raft_consensus.cc:729] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:33.898193 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e07220db7444ae97f294d70eda6a93" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:33.887553 30907 raft_consensus.cc:2726] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240501 20:01:33.862780 30839 consensus_queue.cc:1046] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.039s
I20240501 20:01:33.904801 29773 leader_election.cc:302] T d1d2cac25f354a97b1bfb53d57ae7c08 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:33.913692 29773 leader_election.cc:302] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:33.873622 30815 raft_consensus.cc:2781] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:01:33.868569 30917 raft_consensus.cc:686] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:33.886883 30751 raft_consensus.cc:3037] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:33.850847 30897 raft_consensus.cc:686] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:33.904758 29698 leader_election.cc:302] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:33.899220 30849 raft_consensus.cc:482] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:33.899369 30898 raft_consensus.cc:2726] T 9046e38fc57f4364aa3d539e1fc01928 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:33.926383 29698 leader_election.cc:302] T c651044a305c4168805825a53d910899 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:33.901199 30827 raft_consensus.cc:686] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:33.905381 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece8d4ed84c34b748f7b6272b31c90ac" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:33.953351 29698 leader_election.cc:302] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:33.908491 30759 raft_consensus.cc:482] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:33.893088 30753 consensus_queue.cc:1035] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20240501 20:01:33.893085 30769 consensus_queue.cc:1035] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.057s
I20240501 20:01:33.909950 30912 raft_consensus.cc:3037] T ec073fe645614fe28e7837de3c22510b P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:33.892802 30909 consensus_queue.cc:1046] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
W20240501 20:01:33.905535 30385 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 89476.1us
I20240501 20:01:33.951128 30840 raft_consensus.cc:2726] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:33.975822 29698 leader_election.cc:302] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:33.891047 30842 raft_consensus.cc:504] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:33.906911 30806 consensus_queue.cc:1035] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.050s
I20240501 20:01:33.898823 30886 consensus_queue.cc:1035] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.065s
I20240501 20:01:33.923408 30872 consensus_queue.cc:237] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:33.925926 30907 raft_consensus.cc:2726] T d1d2cac25f354a97b1bfb53d57ae7c08 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:33.912415 30866 raft_consensus.cc:2726] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:01:33.997666 30942 consensus_queue.cc:1046] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:01:33.997486 30907 raft_consensus.cc:3037] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:33.998662 30942 raft_consensus.cc:3032] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:01:33.999019 30942 raft_consensus.cc:729] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 888d99522e4844118f8f1cf26f268c8c, State: Running, Role: LEADER
I20240501 20:01:33.951587 30849 raft_consensus.cc:3037] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:33.999842 30942 consensus_queue.cc:260] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:33.993963 30794 raft_consensus.cc:3032] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 2 LEADER]: Stepping down as leader of term 2
W20240501 20:01:33.980275 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 133453us
I20240501 20:01:33.964330 30761 raft_consensus.cc:2781] T c651044a305c4168805825a53d910899 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:34.001766 30761 raft_consensus.cc:482] T c651044a305c4168805825a53d910899 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:34.001514 30794 raft_consensus.cc:729] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:33.899070 30906 raft_consensus.cc:504] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:33.950824 30815 raft_consensus.cc:686] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 3 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:33.997442 30897 consensus_queue.cc:237] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:34.002056 30761 raft_consensus.cc:3037] T c651044a305c4168805825a53d910899 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:34.002861 30794 consensus_queue.cc:260] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:33.975276 30759 raft_consensus.cc:3037] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:34.003384 30815 consensus_queue.cc:237] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:33.995486 30682 leader_election.cc:288] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:33.949472 30917 consensus_queue.cc:237] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:33.980779 30898 raft_consensus.cc:2781] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:34.001535 30942 raft_consensus.cc:3037] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:33.926203 30384 leader_election.cc:302] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 888d99522e4844118f8f1cf26f268c8c; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:34.006013 29624 leader_election.cc:302] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:33.993734 30935 consensus_queue.cc:260] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:34.010389 30646 raft_consensus.cc:3032] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:34.011153 30921 raft_consensus.cc:3037] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:34.011448 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05df43bdfa479b902b1da297fb5974" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:34.046335 29624 leader_election.cc:302] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:34.040251 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a2bba62f434d1c8489b55c8fec2187" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:34.012238 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfac893b4b4f457694a4eb184b6abb53" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:34.047575 29624 leader_election.cc:302] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:34.048162 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e49dd5180f04313ab33942fb3ddc147" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:34.048645 29624 leader_election.cc:302] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:34.015676 30898 raft_consensus.cc:482] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:34.006057 30829 raft_consensus.cc:504] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:34.016393 29806 raft_consensus.cc:1261] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240501 20:01:34.017197 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6797d82dda2948d8a7445fcd33e44752" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:34.016641 30827 consensus_queue.cc:237] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:34.019500 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f5d1686214485d91a0baf4b870740e" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:34.021075 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f5d1686214485d91a0baf4b870740e" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
W20240501 20:01:34.022135 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 46389.1us
I20240501 20:01:34.058095 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c597f15e844f30837012baed4b0501" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
W20240501 20:01:34.057447 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 143846us
I20240501 20:01:34.059192 29698 leader_election.cc:398] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Vote denied by peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959) with higher term. Message: Invalid argument: T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d for earlier term 2. Current term is 3.
I20240501 20:01:34.064107 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca3948261374cef8b1b206a43c00173" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:34.064945 29735 raft_consensus.cc:2454] T 1ca3948261374cef8b1b206a43c00173 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
I20240501 20:01:34.066375 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e49dd5180f04313ab33942fb3ddc147" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:34.067559 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a35893b91247a99912eba787d787cf" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:34.037765 29656 raft_consensus.cc:1226] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:34.071486 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48630ae6eb394ac1b54955dd01cf0ed2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:34.072405 29656 raft_consensus.cc:2379] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d in current term 2: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:34.073276 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010ceabf3e5a4b1c9330b80dade765bb" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:34.006129 30794 raft_consensus.cc:3037] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:34.053061 29809 raft_consensus.cc:1261] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20240501 20:01:34.075122 29773 leader_election.cc:334] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 2.418s (SENT)
I20240501 20:01:34.045840 30933 raft_consensus.cc:2726] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240501 20:01:34.054253 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649b71f9f267437ca51c758dd9ca6987" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:34.050155 30898 raft_consensus.cc:3037] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:34.054309 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36057b3f24d44b9aac3a1378053a395f" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:34.056015 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:34.035197 30800 raft_consensus.cc:2726] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20240501 20:01:34.068853 29772 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 196512us
I20240501 20:01:34.083228 29772 leader_election.cc:302] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters: fdfeac4190e045f290e003b2298d988d
I20240501 20:01:34.036278 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a2bba62f434d1c8489b55c8fec2187" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:34.083966 29734 raft_consensus.cc:2454] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 3.
I20240501 20:01:34.084435 29772 leader_election.cc:302] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:34.084794 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec073fe645614fe28e7837de3c22510b" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:34.085511 29772 leader_election.cc:302] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:34.085808 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8edcd903224d48f889b46e1b5318e0ec" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:34.077368 29698 leader_election.cc:302] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:34.012293 30946 consensus_queue.cc:1035] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20240501 20:01:34.078006 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5525fc4b82a44b0991c3dd6098fde81" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:34.079278 29659 raft_consensus.cc:2379] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d in current term 4: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:34.054811 30935 raft_consensus.cc:3037] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:34.093214 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5525f546426c47a38edf7f56ad04b7ab" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:34.074147 29656 raft_consensus.cc:2379] T 010ceabf3e5a4b1c9330b80dade765bb P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d in current term 2: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:34.052309 29802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9472c53f79ec423f82c396bd980674e2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:34.044725 30646 raft_consensus.cc:729] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:34.087095 29733 raft_consensus.cc:2454] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 3.
I20240501 20:01:34.118402 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8cd40cb4044d18a86001ce6ddf97ab" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:34.119129 29733 raft_consensus.cc:2454] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
I20240501 20:01:34.120038 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "456035a1cb3f4f7da52d0fc1dd1c656c" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:34.121160 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0374bb790ed345ed90452595f71383a2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:34.089843 29772 leader_election.cc:398] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Vote denied by peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959) with higher term. Message: Invalid argument: T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d for earlier term 2. Current term is 3.
I20240501 20:01:34.122493 29772 leader_election.cc:401] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20240501 20:01:34.116022 30894 raft_consensus.cc:2781] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:34.074992 30937 raft_consensus.cc:2726] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 20:01:34.050493 30839 consensus_queue.cc:1035] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.078s
I20240501 20:01:34.062881 30769 raft_consensus.cc:2781] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:34.111299 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ed27033ab1645e9ab0600ab635d9589" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:34.091809 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:34.116459 30866 raft_consensus.cc:2726] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 20:01:34.101991 30954 raft_consensus.cc:2741] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader election decision vote started in defunct term 3: won
I20240501 20:01:34.117128 29735 raft_consensus.cc:2396] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240501 20:01:34.054189 30649 consensus_queue.cc:1046] T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.048s
W20240501 20:01:34.068675 30383 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 207967us
I20240501 20:01:34.137527 29657 raft_consensus.cc:2379] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d in current term 2: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:34.127477 30894 raft_consensus.cc:482] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting leader election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
W20240501 20:01:34.057443 30384 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 144487us
I20240501 20:01:34.094904 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e9f75387564275a60d38e9a910319c" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:34.142853 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac2171683884f6f97cdfeb0a656cb49" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:34.136777 29728 raft_consensus.cc:2354] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d for earlier term 3. Current term is 4.
I20240501 20:01:34.139376 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3f29f0eeea4b119984634202e5d9cd" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:34.143244 29731 raft_consensus.cc:2354] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f for earlier term 3. Current term is 4.
I20240501 20:01:34.143244 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010ceabf3e5a4b1c9330b80dade765bb" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:34.158985 29733 raft_consensus.cc:3037] T 010ceabf3e5a4b1c9330b80dade765bb P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:34.101670 30753 consensus_queue.cc:1035] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
I20240501 20:01:34.154100 29656 raft_consensus.cc:2379] T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d in current term 2: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:34.088258 29698 leader_election.cc:398] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Vote denied by peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959) with higher term. Message: Invalid argument: T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d for earlier term 2. Current term is 3.
I20240501 20:01:34.293570 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5345f6eb62449ba69b4a9c2e349b31" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
W20240501 20:01:34.350087 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 262041us
I20240501 20:01:34.352170 29698 leader_election.cc:302] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:34.353267 29698 leader_election.cc:302] T 1ca3948261374cef8b1b206a43c00173 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:34.354256 29698 leader_election.cc:302] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:34.141608 30769 raft_consensus.cc:482] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:34.249768 29729 raft_consensus.cc:2454] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 2.
W20240501 20:01:34.279538 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 230953us
I20240501 20:01:34.141407 30951 raft_consensus.cc:2726] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240501 20:01:34.226294 29657 raft_consensus.cc:1226] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 2. Current term is 3. Ops: []
I20240501 20:01:34.159811 30954 raft_consensus.cc:2726] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959) with higher term. Message: Invalid argument: T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d for earlier term 2. Current term is 3.
I20240501 20:01:34.139688 30809 raft_consensus.cc:2781] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:01:34.224934 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d88139f89c47ff84cad674147aae0f" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:34.235775 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f261d2a05bf4c838532049601694b4a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:34.367805 29730 raft_consensus.cc:2454] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 3.
I20240501 20:01:34.126427 30836 consensus_queue.cc:1035] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20240501 20:01:34.258169 30829 leader_election.cc:288] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:34.279137 29658 raft_consensus.cc:1261] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240501 20:01:34.223127 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f5d1686214485d91a0baf4b870740e" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:34.334204 29806 raft_consensus.cc:1261] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20240501 20:01:34.376632 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49da4246f5f6438abe0e7e8c9a523d80" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:34.209333 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5525fc4b82a44b0991c3dd6098fde81" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:34.261145 30842 leader_election.cc:288] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:34.291715 29805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41858 (request call id 800) took 1110 ms. Trace:
I20240501 20:01:34.241130 30761 raft_consensus.cc:504] T c651044a305c4168805825a53d910899 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:34.210150 30646 consensus_queue.cc:260] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:34.157284 29651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07bdb8b09b8c4ef4b8e49daa9b8c01d6" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:34.152022 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8787bc4d6b44649b9138d906730110a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:34.157461 30894 raft_consensus.cc:3037] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:34.392959 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649b71f9f267437ca51c758dd9ca6987" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:34.391456 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649b71f9f267437ca51c758dd9ca6987" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:34.398823 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6797d82dda2948d8a7445fcd33e44752" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:34.307632 30713 raft_consensus.cc:3032] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:34.400234 30713 raft_consensus.cc:729] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:34.401265 30713 consensus_queue.cc:260] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:34.402468 30713 raft_consensus.cc:3037] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:01:34.403180 30949 log.cc:924] Time spent T 2d86f8848a184a87947fd927c44e5a35 P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 0.167s user 0.000s sys 0.001s
I20240501 20:01:34.335654 30852 raft_consensus.cc:3037] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Advancing to term 4
W20240501 20:01:34.350008 30961 log.cc:924] Time spent T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c: Append to log took a long time: real 0.081s user 0.002s sys 0.000s
I20240501 20:01:34.371186 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d86f8848a184a87947fd927c44e5a35" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:34.419153 29732 raft_consensus.cc:2379] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 2: Already voted for candidate ee0d97247d394a8facfa774f15f3a67d in this term.
I20240501 20:01:34.373060 29624 leader_election.cc:302] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:34.358469 30753 raft_consensus.cc:2726] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:01:34.359503 30769 raft_consensus.cc:3037] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:34.357780 30917 raft_consensus.cc:2726] T 1ca3948261374cef8b1b206a43c00173 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:34.361882 30897 raft_consensus.cc:2726] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:34.378011 29735 raft_consensus.cc:2362] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 3. Re-sending same reply.
I20240501 20:01:34.429962 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:34.431397 29735 raft_consensus.cc:1226] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20240501 20:01:34.430967 30912 raft_consensus.cc:504] T ec073fe645614fe28e7837de3c22510b P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:34.379093 29773 leader_election.cc:334] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 1.524s (SENT)
W20240501 20:01:34.432255 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 53198.3us
I20240501 20:01:34.357375 30906 leader_election.cc:288] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:34.436450 29732 raft_consensus.cc:1226] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20240501 20:01:34.377405 29806 raft_consensus.cc:2454] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 3.
I20240501 20:01:34.349932 30759 raft_consensus.cc:504] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:34.390884 30384 leader_election.cc:302] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 888d99522e4844118f8f1cf26f268c8c, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:34.381246 30809 raft_consensus.cc:686] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [term 2 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:34.468911 30809 consensus_queue.cc:237] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:34.472306 29728 raft_consensus.cc:2379] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 2: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:34.472493 30824 raft_consensus.cc:2781] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:34.473167 30824 raft_consensus.cc:482] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Starting leader election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:34.473664 30824 raft_consensus.cc:3037] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:34.466724 29700 leader_election.cc:302] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:34.474493 29726 raft_consensus.cc:2354] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d for earlier term 3. Current term is 4.
I20240501 20:01:34.366993 29802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:34.369230 30966 raft_consensus.cc:3032] T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Stepping down as leader of term 1
W20240501 20:01:34.412889 30947 log.cc:924] Time spent T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.106s user 0.000s sys 0.001s
I20240501 20:01:34.424317 30751 raft_consensus.cc:2726] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:01:34.446931 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "456035a1cb3f4f7da52d0fc1dd1c656c" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:34.454566 29807 raft_consensus.cc:1261] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20240501 20:01:34.455638 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8787bc4d6b44649b9138d906730110a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:34.458354 29733 raft_consensus.cc:2454] T 010ceabf3e5a4b1c9330b80dade765bb P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 2.
I20240501 20:01:34.356035 29698 leader_election.cc:302] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
W20240501 20:01:34.467857 30384 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 81272.7us
I20240501 20:01:34.496929 29733 raft_consensus.cc:1226] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 2. Current term is 3. Ops: []
I20240501 20:01:34.486882 30894 raft_consensus.cc:504] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } }
I20240501 20:01:34.491753 29803 raft_consensus.cc:1226] T ec073fe645614fe28e7837de3c22510b P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 2. Current term is 3. Ops: []
I20240501 20:01:34.498283 29733 raft_consensus.cc:1226] T 010ceabf3e5a4b1c9330b80dade765bb P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:34.498944 30894 leader_election.cc:288] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 election: Requested vote from peers ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861), 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959)
I20240501 20:01:34.499578 29733 raft_consensus.cc:1226] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:34.499955 29803 raft_consensus.cc:1261] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20240501 20:01:34.501533 29733 raft_consensus.cc:1226] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:34.421206 30646 raft_consensus.cc:3037] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:34.436955 30937 raft_consensus.cc:2781] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:01:34.508373 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69915dd5992641e3a24f7f45f1cd5c0c" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
W20240501 20:01:34.475462 30957 log.cc:924] Time spent T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
I20240501 20:01:34.500902 29735 raft_consensus.cc:1226] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20240501 20:01:34.503433 29727 raft_consensus.cc:2379] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 2: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:34.511376 29726 raft_consensus.cc:1226] T c651044a305c4168805825a53d910899 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 1. Current term is 2. Ops: []
W20240501 20:01:34.511919 29772 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 422177us
W20240501 20:01:34.503206 30963 log.cc:924] Time spent T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d: Append to log took a long time: real 0.115s user 0.001s sys 0.000s
I20240501 20:01:34.515731 29731 raft_consensus.cc:1226] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20240501 20:01:34.492772 30886 raft_consensus.cc:2781] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:34.494184 30966 raft_consensus.cc:729] T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:34.518818 29772 leader_election.cc:302] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:34.494802 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcf30cf66714efdac997862dafbe4a2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:34.489712 30897 consensus_queue.cc:1035] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20240501 20:01:34.528729 29772 leader_election.cc:302] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:34.518652 30906 raft_consensus.cc:2781] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:01:34.519053 30886 raft_consensus.cc:482] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:34.495100 30898 raft_consensus.cc:504] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:34.517858 30937 raft_consensus.cc:686] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f [term 2 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:34.523674 30759 leader_election.cc:288] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:34.385712 29805 rpcz_store.cc:276] 0501 20:01:33.181545 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:33.760637 (+579092us) service_pool.cc:225] Handling call
0501 20:01:33.808148 (+ 47511us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:33.824753 (+ 16605us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:33.829359 (+ 4606us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:33.840286 (+ 10927us) raft_consensus.cc:1631] Marking committed up to 1
0501 20:01:33.846020 (+ 5734us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:34.291450 (+445430us) spinlock_profiling.cc:243] Waited 6.29 ms on lock 0x7b54001491d4. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787c1a1 00007fe45787abee 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc 00007fe455e9e782 00007fe455ca2bcc 00007fe455ca2006
0501 20:01:34.291505 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:34.291652 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14463232}
I20240501 20:01:34.537760 30415 raft_consensus.cc:1226] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 4 LEADER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 3. Current term is 4. Ops: []
I20240501 20:01:34.517357 30836 consensus_queue.cc:1035] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20240501 20:01:34.529394 30751 raft_consensus.cc:2726] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:01:34.538827 30886 raft_consensus.cc:3037] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:34.542220 30761 leader_election.cc:288] T c651044a305c4168805825a53d910899 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:34.542521 30906 raft_consensus.cc:686] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [term 3 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
W20240501 20:01:34.574532 30907 consensus_meta.cc:287] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d: Time spent flushing consensus metadata: real 0.574s user 0.009s sys 0.001s
I20240501 20:01:34.575147 30907 raft_consensus.cc:2726] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:34.558544 30966 consensus_queue.cc:260] T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:34.580626 30966 raft_consensus.cc:3037] T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:01:34.587853 30942 consensus_meta.cc:287] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c: Time spent flushing consensus metadata: real 0.556s user 0.007s sys 0.002s
W20240501 20:01:34.585916 30969 log.cc:924] Time spent T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c: Append to log took a long time: real 0.199s user 0.001s sys 0.000s
I20240501 20:01:34.577772 30937 consensus_queue.cc:237] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:34.420186 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 47242.1us
I20240501 20:01:34.618526 29624 leader_election.cc:302] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:34.619449 29727 raft_consensus.cc:1226] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 2. Current term is 3. Ops: []
I20240501 20:01:34.620759 29727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 724) took 1039 ms. Trace:
W20240501 20:01:34.621222 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 114907us
I20240501 20:01:34.621306 29727 rpcz_store.cc:276] 0501 20:01:33.581146 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:34.617985 (+1036839us) service_pool.cc:225] Handling call
0501 20:01:34.618878 (+ 893us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:34.620352 (+ 1474us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:34.620662 (+ 310us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:34.623848 29655 raft_consensus.cc:2396] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240501 20:01:34.624724 29727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 725) took 1043 ms. Trace:
I20240501 20:01:34.625422 29726 raft_consensus.cc:1261] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240501 20:01:34.626345 29726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 793) took 1047 ms. Trace:
I20240501 20:01:34.626842 29726 rpcz_store.cc:276] 0501 20:01:33.578617 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:34.512108 (+933491us) service_pool.cc:225] Handling call
0501 20:01:34.624770 (+112662us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:34.625978 (+ 1208us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:34.626254 (+ 276us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240501 20:01:34.506421 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 150435us
I20240501 20:01:34.627723 29624 leader_election.cc:302] T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:34.629892 29698 leader_election.cc:302] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:34.630501 29624 leader_election.cc:302] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:34.632691 29733 raft_consensus.cc:1226] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20240501 20:01:34.633452 29731 raft_consensus.cc:1226] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:34.634256 29731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 723) took 1053 ms. Trace:
I20240501 20:01:34.634672 29731 rpcz_store.cc:276] 0501 20:01:33.580890 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:34.519748 (+938858us) service_pool.cc:225] Handling call
0501 20:01:34.632928 (+113180us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:34.633949 (+ 1021us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:34.634179 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:34.635226 29624 leader_election.cc:302] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
W20240501 20:01:34.638406 29772 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 110933us
I20240501 20:01:34.642769 30414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e07220db7444ae97f294d70eda6a93" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c" is_pre_election: true
I20240501 20:01:34.643442 30751 raft_consensus.cc:2726] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:01:34.653203 30935 raft_consensus.cc:2781] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:01:34.654208 30935 raft_consensus.cc:686] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
W20240501 20:01:34.655159 30921 consensus_meta.cc:287] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d: Time spent flushing consensus metadata: real 0.605s user 0.011s sys 0.002s
I20240501 20:01:34.656775 30965 raft_consensus.cc:2781] T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:01:34.657598 30965 raft_consensus.cc:686] T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f [term 2 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:34.658041 30839 raft_consensus.cc:2781] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:01:34.658756 30852 raft_consensus.cc:2726] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:01:34.658876 30839 raft_consensus.cc:686] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [term 5 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:34.578040 30912 leader_election.cc:288] T ec073fe645614fe28e7837de3c22510b P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:34.655867 30921 raft_consensus.cc:2726] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:01:34.625221 29727 rpcz_store.cc:276] 0501 20:01:33.581396 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:34.623253 (+1041857us) service_pool.cc:225] Handling call
0501 20:01:34.623852 (+ 599us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:34.624251 (+ 399us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:34.624290 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:34.624354 (+ 64us) raft_consensus.cc:1631] Marking committed up to 4
0501 20:01:34.624409 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:34.624491 (+ 82us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:34.624642 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:34.666465 30839 consensus_queue.cc:237] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:34.697130 30894 consensus_queue.cc:1035] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 20:01:34.698163 30897 raft_consensus.cc:2726] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:34.659286 30965 consensus_queue.cc:237] T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:34.704466 30794 consensus_meta.cc:287] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d: Time spent flushing consensus metadata: real 0.626s user 0.008s sys 0.003s
I20240501 20:01:34.706480 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c651044a305c4168805825a53d910899" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:34.710078 29802 raft_consensus.cc:1261] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20240501 20:01:34.711407 30650 raft_consensus.cc:482] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:34.710003 30898 leader_election.cc:288] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:34.722929 29726 raft_consensus.cc:1226] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:34.725153 30966 consensus_queue.cc:1046] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 20:01:34.712152 30650 raft_consensus.cc:504] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:34.726361 30966 raft_consensus.cc:3032] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:01:34.726816 30966 raft_consensus.cc:729] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:34.726840 30906 consensus_queue.cc:237] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:34.729496 30849 consensus_meta.cc:287] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d: Time spent flushing consensus metadata: real 0.729s user 0.011s sys 0.002s
I20240501 20:01:34.729946 29726 raft_consensus.cc:1261] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Refusing update from remote peer 888d99522e4844118f8f1cf26f268c8c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240501 20:01:34.730000 30849 raft_consensus.cc:504] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:34.731609 29801 raft_consensus.cc:1226] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20240501 20:01:34.731757 29726 raft_consensus.cc:3032] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Stepping down as leader of term 4
I20240501 20:01:34.733568 29726 raft_consensus.cc:729] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:34.732755 29731 raft_consensus.cc:1226] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:34.743752 29732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 708) took 1184 ms. Trace:
I20240501 20:01:34.746378 29726 consensus_queue.cc:260] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:34.744283 29732 rpcz_store.cc:276] 0501 20:01:33.558824 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:34.437383 (+878559us) service_pool.cc:225] Handling call
0501 20:01:34.437701 (+ 318us) raft_consensus.cc:1457] Updating replica for 1 ops
0501 20:01:34.438074 (+ 373us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:34.438113 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0501 20:01:34.491101 (+ 52988us) log.cc:842] Serialized 26 byte log entry
0501 20:01:34.632579 (+141478us) raft_consensus.cc:1631] Marking committed up to 1
0501 20:01:34.632682 (+ 103us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:34.632819 (+ 137us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0501 20:01:34.743354 (+110535us) raft_consensus.cc:1671] finished
0501 20:01:34.743382 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:34.743561 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1529,"raft.queue_time_us":1688,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":26943,"thread_start_us":127571,"threads_started":1,"wal-append.queue_time_us":236983}
I20240501 20:01:34.751513 29726 raft_consensus.cc:3037] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:34.752775 29733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 720) took 1175 ms. Trace:
I20240501 20:01:34.765758 29733 rpcz_store.cc:276] 0501 20:01:33.580032 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:34.503078 (+923046us) service_pool.cc:225] Handling call
0501 20:01:34.508229 (+ 5151us) raft_consensus.cc:1457] Updating replica for 1 ops
0501 20:01:34.633325 (+125096us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:34.752383 (+119058us) spinlock_profiling.cc:243] Waited 124 ms on lock 0x7b54001013d4. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787c1a1 00007fe45787abee 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc 00007fe455e9e782 00007fe455ca2bcc 00007fe455ca2006
0501 20:01:34.752688 (+ 305us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":284402944}
I20240501 20:01:34.770694 29558 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:43396 (request call id 161) took 6204 ms. Trace:
I20240501 20:01:34.772882 30836 consensus_queue.cc:1035] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 20:01:34.756490 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e49dd5180f04313ab33942fb3ddc147" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:34.775975 29805 raft_consensus.cc:3037] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:34.771296 29558 rpcz_store.cc:276] 0501 20:01:28.566615 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:28.601836 (+ 35221us) service_pool.cc:225] Handling call
0501 20:01:34.770359 (+6168523us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0501 20:01:33.539654 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 20:01:33.570238 (+ 30584us) write_op.cc:424] Acquiring schema lock in shared mode
0501 20:01:33.572554 (+ 2316us) write_op.cc:427] Acquired schema lock
0501 20:01:33.574781 (+ 2227us) tablet.cc:660] Decoding operations
0501 20:01:33.594466 (+ 19685us) write_op.cc:612] Acquiring the partition lock for write op
0501 20:01:33.600205 (+ 5739us) write_op.cc:633] Partition lock acquired for write op
0501 20:01:33.602553 (+ 2348us) tablet.cc:683] Acquiring locks for 4 operations
0501 20:01:33.609421 (+ 6868us) tablet.cc:699] Row locks acquired
0501 20:01:33.609466 (+ 45us) write_op.cc:260] PREPARE: finished
0501 20:01:33.609655 (+ 189us) write_op.cc:270] Start()
0501 20:01:33.609801 (+ 146us) write_op.cc:276] Timestamp: P: 1714593693609599 usec, L: 0
0501 20:01:33.609831 (+ 30us) op_driver.cc:331] REPLICATION: starting
0501 20:01:33.610601 (+ 770us) log.cc:842] Serialized 1717 byte log entry
0501 20:01:33.858724 (+248123us) op_driver.cc:444] REPLICATION: finished
0501 20:01:34.217374 (+358650us) write_op.cc:301] APPLY: starting
0501 20:01:34.226977 (+ 9603us) tablet.cc:1365] starting BulkCheckPresence
0501 20:01:34.247016 (+ 20039us) tablet.cc:1368] finished BulkCheckPresence
0501 20:01:34.247702 (+ 686us) tablet.cc:1370] starting ApplyRowOperation cycle
0501 20:01:34.255217 (+ 7515us) tablet.cc:1381] finished ApplyRowOperation cycle
0501 20:01:34.255424 (+ 207us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0501 20:01:34.255488 (+ 64us) write_op.cc:312] APPLY: finished
0501 20:01:34.259559 (+ 4071us) log.cc:842] Serialized 40 byte log entry
0501 20:01:34.259778 (+ 219us) write_op.cc:481] Releasing partition, row and schema locks
0501 20:01:34.259900 (+ 122us) write_op.cc:446] Released schema lock
0501 20:01:34.259952 (+ 52us) write_op.cc:341] FINISH: Updating metrics
0501 20:01:34.263480 (+ 3528us) write_op.cc:481] Releasing partition, row and schema locks
0501 20:01:34.263522 (+ 42us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":537,"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":277853,"apply.run_cpu_time_us":3695,"apply.run_wall_time_us":54691,"num_ops":4,"prepare.queue_time_us":340644,"prepare.run_cpu_time_us":25839,"prepare.run_wall_time_us":242946,"raft.queue_time_us":27867,"raft.run_cpu_time_us":29808,"raft.run_wall_time_us":268021,"replication_time_us":241399,"thread_start_us":515361,"threads_started":4,"wal-append.queue_time_us":164140}]]}
I20240501 20:01:34.780450 30650 leader_election.cc:288] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:34.787304 29624 leader_election.cc:302] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:34.788555 29729 raft_consensus.cc:2396] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20240501 20:01:34.788946 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 103265us
I20240501 20:01:34.790388 29726 raft_consensus.cc:1261] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Refusing update from remote peer 888d99522e4844118f8f1cf26f268c8c: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240501 20:01:34.790328 29729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 787) took 1226 ms. Trace:
I20240501 20:01:34.794766 29729 rpcz_store.cc:276] 0501 20:01:33.563581 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:34.400006 (+836425us) service_pool.cc:225] Handling call
0501 20:01:34.789992 (+389986us) spinlock_profiling.cc:243] Waited 191 ms on lock 0x7b54003705d4. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787f4fe 00007fe45c7bd9d7 00007fe455ea1c14 00007fe455ea1ba2 00007fe455ea1ad1 00007fe455ea1a4c 00007fe455ea0d02 00007fe455ca2bcc 00007fe455ca2006 00007fe455ca19af
0501 20:01:34.790193 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":440281472}
I20240501 20:01:34.744210 30829 consensus_queue.cc:1046] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.053s
W20240501 20:01:34.797371 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 168057us
I20240501 20:01:34.798173 30829 raft_consensus.cc:3032] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:01:34.798650 30829 raft_consensus.cc:729] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:34.798905 29698 leader_election.cc:302] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:34.791013 30650 raft_consensus.cc:2726] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240501 20:01:34.668716 30935 consensus_queue.cc:237] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:34.791512 30944 consensus_queue.cc:1035] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240501 20:01:34.791996 30849 leader_election.cc:288] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:34.735162 29772 leader_election.cc:302] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:34.804276 30751 raft_consensus.cc:3037] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:34.800452 30980 raft_consensus.cc:2781] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election won for term 3
W20240501 20:01:34.804636 29772 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 137286us
I20240501 20:01:34.804966 30980 raft_consensus.cc:482] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting leader election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:34.805519 30980 raft_consensus.cc:3037] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Advancing to term 3
W20240501 20:01:34.806907 30953 log.cc:924] Time spent T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 0.342s user 0.003s sys 0.000s
I20240501 20:01:34.808712 30815 consensus_queue.cc:1046] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 20:01:34.809881 30815 raft_consensus.cc:3032] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:01:34.810384 30815 raft_consensus.cc:729] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:34.812526 29801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38cbeb0dc0c14e3d9182101eea885625" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:34.813406 29625 leader_election.cc:302] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:34.814287 30414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e89c0572e4ee7be8e5315b459bdb3" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c" is_pre_election: true
I20240501 20:01:34.813743 30769 raft_consensus.cc:504] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:34.816715 30794 consensus_queue.cc:1046] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 20:01:34.819062 29624 leader_election.cc:398] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Vote denied by peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907) with higher term. Message: Invalid argument: T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f for earlier term 3. Current term is 4.
I20240501 20:01:34.819406 30761 raft_consensus.cc:482] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:34.820077 30761 raft_consensus.cc:504] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:34.820945 30753 raft_consensus.cc:3032] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:01:34.821386 30753 raft_consensus.cc:729] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:34.822158 30761 leader_election.cc:288] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:34.822516 30753 consensus_queue.cc:260] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:34.823765 30650 raft_consensus.cc:2781] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:01:34.825757 30650 raft_consensus.cc:686] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:34.827194 30650 consensus_queue.cc:237] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:34.818935 30966 consensus_queue.cc:260] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:34.811844 30815 consensus_queue.cc:260] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:34.819432 29624 leader_election.cc:401] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20240501 20:01:34.832353 29698 leader_election.cc:302] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:34.844350 30808 consensus_queue.cc:1035] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
I20240501 20:01:34.844772 29800 raft_consensus.cc:1261] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20240501 20:01:34.834563 30753 raft_consensus.cc:3037] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:34.853783 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5525f546426c47a38edf7f56ad04b7ab" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:34.840937 30769 leader_election.cc:288] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:34.857985 30886 raft_consensus.cc:504] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:34.858520 30794 consensus_queue.cc:1035] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:34.859555 30851 consensus_queue.cc:1046] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 20:01:34.860978 30894 raft_consensus.cc:2726] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 20:01:34.863382 30810 consensus_queue.cc:1046] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 20:01:34.799957 30829 consensus_queue.cc:260] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:34.879776 29734 raft_consensus.cc:2454] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 3.
W20240501 20:01:34.880023 30831 consensus_meta.cc:287] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f: Time spent flushing consensus metadata: real 0.978s user 0.012s sys 0.004s
I20240501 20:01:34.848858 30852 consensus_queue.cc:1035] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
W20240501 20:01:34.884496 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 53518.6us
W20240501 20:01:34.887696 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 69302.8us
I20240501 20:01:34.880807 30815 raft_consensus.cc:3037] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:34.886245 30851 raft_consensus.cc:3032] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:01:34.884967 30800 raft_consensus.cc:2726] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907) with higher term. Message: Invalid argument: T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f for earlier term 3. Current term is 4.
I20240501 20:01:34.881016 30944 consensus_queue.cc:1035] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.031s
I20240501 20:01:34.910369 30751 raft_consensus.cc:2726] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:01:34.912832 30851 raft_consensus.cc:729] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:34.894340 30824 raft_consensus.cc:504] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:34.930472 29728 raft_consensus.cc:1226] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [term 3 LEADER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20240501 20:01:34.909721 30831 raft_consensus.cc:504] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:34.948127 30415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "696121417ea9403990456cfc6fc4f416" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c" is_pre_election: true
I20240501 20:01:34.949637 30415 raft_consensus.cc:2396] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20240501 20:01:34.951134 29654 raft_consensus.cc:1226] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20240501 20:01:34.951346 30831 leader_election.cc:288] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:34.952852 30414 raft_consensus.cc:1226] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [term 2 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:34.957435 29624 leader_election.cc:302] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:34.952844 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649b71f9f267437ca51c758dd9ca6987" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:34.955204 29658 raft_consensus.cc:1261] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Refusing update from remote peer 888d99522e4844118f8f1cf26f268c8c: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240501 20:01:34.946697 29805 raft_consensus.cc:2454] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 4.
I20240501 20:01:34.968122 29806 raft_consensus.cc:1226] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 2. Current term is 3. Ops: []
W20240501 20:01:34.968128 30713 consensus_meta.cc:287] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f: Time spent flushing consensus metadata: real 0.544s user 0.007s sys 0.011s
W20240501 20:01:34.970748 29773 leader_election.cc:334] T f6d88139f89c47ff84cad674147aae0f P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 2.238s (SENT)
I20240501 20:01:34.971351 29773 leader_election.cc:302] T f6d88139f89c47ff84cad674147aae0f P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
W20240501 20:01:34.975574 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 58748us
I20240501 20:01:34.976593 29698 leader_election.cc:302] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:34.977571 30753 raft_consensus.cc:2726] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 20:01:34.978111 29654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59136 (request call id 703) took 1571 ms. Trace:
I20240501 20:01:34.978991 29700 leader_election.cc:302] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:34.978698 29654 rpcz_store.cc:276] 0501 20:01:33.406806 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:33.692337 (+285531us) service_pool.cc:225] Handling call
0501 20:01:33.692709 (+ 372us) raft_consensus.cc:1457] Updating replica for 1 ops
0501 20:01:34.951774 (+1259065us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:34.977792 (+ 26018us) spinlock_profiling.cc:243] Waited 1.26 s on lock 0x7b5400132c54. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787c1a1 00007fe45787abee 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc 00007fe455e9e782 00007fe455ca2bcc 00007fe455ca2006
0501 20:01:34.978040 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2892906368}
I20240501 20:01:34.980288 30794 raft_consensus.cc:2781] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:01:34.980926 30794 raft_consensus.cc:686] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:34.982007 29655 raft_consensus.cc:1261] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Refusing update from remote peer 888d99522e4844118f8f1cf26f268c8c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20240501 20:01:34.982656 30983 log.cc:924] Time spent T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 0.121s user 0.002s sys 0.000s
I20240501 20:01:34.982162 30794 consensus_queue.cc:237] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:34.978049 29698 leader_election.cc:302] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:34.990130 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f5d1686214485d91a0baf4b870740e" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:34.990423 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e89c0572e4ee7be8e5315b459bdb3" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:34.991114 29800 raft_consensus.cc:2454] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 4.
I20240501 20:01:34.991654 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49da4246f5f6438abe0e7e8c9a523d80" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:34.993631 30829 raft_consensus.cc:3037] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:34.997272 30824 leader_election.cc:288] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 2 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:34.998028 29772 consensus_peers.cc:485] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d -> Peer 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227): Couldn't send request to peer 888d99522e4844118f8f1cf26f268c8c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77be7101c0fe4085b6706296f609870a. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:34.999538 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c651044a305c4168805825a53d910899" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:35.002136 29805 raft_consensus.cc:1226] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 1. Current term is 2. Ops: []
W20240501 20:01:35.003540 30984 log.cc:924] Time spent T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.125s user 0.001s sys 0.000s
I20240501 20:01:35.004101 30851 consensus_queue.cc:260] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:35.005630 30851 raft_consensus.cc:3037] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:35.007750 30980 raft_consensus.cc:504] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:35.010005 30980 leader_election.cc:288] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:35.016968 30980 consensus_queue.cc:1046] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:35.007788 29558 catalog_manager.cc:5484] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c reported cstate change: term changed from 3 to 4, leader changed from 4c8da1d580614449a636b855bf11b95f (127.27.217.193) to 888d99522e4844118f8f1cf26f268c8c (127.27.217.196). New cstate: current_term: 4 leader_uuid: "888d99522e4844118f8f1cf26f268c8c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 20:01:35.012169 30982 log.cc:924] Time spent T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 0.135s user 0.000s sys 0.002s
I20240501 20:01:35.012687 30921 raft_consensus.cc:2781] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:35.013159 29807 raft_consensus.cc:3037] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:35.013584 30933 consensus_queue.cc:1035] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240501 20:01:35.014484 29698 leader_election.cc:302] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:35.015501 29728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 710) took 1456 ms. Trace:
I20240501 20:01:35.017297 30965 raft_consensus.cc:3032] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Stepping down as leader of term 3
W20240501 20:01:35.018540 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 101658us
I20240501 20:01:35.010725 29625 leader_election.cc:302] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters: 888d99522e4844118f8f1cf26f268c8c
I20240501 20:01:35.024902 30809 raft_consensus.cc:2781] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240501 20:01:34.971885 30886 leader_election.cc:288] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:35.018968 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36057b3f24d44b9aac3a1378053a395f" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:35.019143 30650 raft_consensus.cc:2781] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:01:34.984988 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8edcd903224d48f889b46e1b5318e0ec" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:35.033587 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3f29f0eeea4b119984634202e5d9cd" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:35.042730 30886 raft_consensus.cc:2781] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:35.053494 30886 raft_consensus.cc:482] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting leader election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:35.053434 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e38a2e91a3d48d09a22abce723af833" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:35.054026 30886 raft_consensus.cc:3037] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:35.041834 30851 raft_consensus.cc:482] T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:35.055203 30851 raft_consensus.cc:504] T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:35.020613 30751 raft_consensus.cc:2726] T f6d88139f89c47ff84cad674147aae0f P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:35.057682 29734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 791) took 1479 ms. Trace:
I20240501 20:01:35.058100 29734 rpcz_store.cc:276] 0501 20:01:33.578062 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:34.085595 (+507533us) service_pool.cc:225] Handling call
0501 20:01:35.057416 (+971821us) spinlock_profiling.cc:243] Waited 605 ms on lock 0x7b54001077d4. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787f4fe 00007fe45c7bd9d7 00007fe455ea1c14 00007fe455ea1ba2 00007fe455ea1ad1 00007fe455ea1a4c 00007fe455ea0d02 00007fe455ca2bcc 00007fe455ca2006 00007fe455ca19af
0501 20:01:35.057602 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1392201600}
I20240501 20:01:35.059749 29735 raft_consensus.cc:1226] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20240501 20:01:35.060597 29624 leader_election.cc:302] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
W20240501 20:01:35.034075 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 87860us
I20240501 20:01:35.023453 30921 raft_consensus.cc:482] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting leader election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
W20240501 20:01:35.035681 30977 log.cc:924] Time spent T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.159s user 0.002s sys 0.000s
I20240501 20:01:35.075122 30851 leader_election.cc:288] T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:35.075640 30921 raft_consensus.cc:3037] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:35.023542 29728 rpcz_store.cc:276] 0501 20:01:33.559203 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:34.473196 (+913993us) service_pool.cc:225] Handling call
0501 20:01:34.763866 (+290670us) spinlock_profiling.cc:243] Waited 16.5 ms on lock 0x7b58001b3098. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe454797b51 00007fe4549193b1 00007fe454919262 00007fe45d80b889 00007fe45c7483c0 00007fe45c80e3bb 00007fe45c80e441 00007fe45c7d2060 00007fe45c7bd5a9 00007fe45c7bd052 00007fe455e9f694
0501 20:01:34.772551 (+ 8685us) raft_consensus.cc:1457] Updating replica for 1 ops
0501 20:01:34.946495 (+173944us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:35.015208 (+ 68713us) spinlock_profiling.cc:243] Waited 157 ms on lock 0x7b540020f8d4. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787c1a1 00007fe45787abee 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc 00007fe455e9e782 00007fe455ca2bcc 00007fe455ca2006
0501 20:01:35.015418 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":399940736}
I20240501 20:01:35.024101 30965 raft_consensus.cc:729] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:35.040663 30753 raft_consensus.cc:3032] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:01:35.085893 29807 raft_consensus.cc:2454] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 4.
I20240501 20:01:35.087193 29735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 721) took 1506 ms. Trace:
I20240501 20:01:35.025290 30809 raft_consensus.cc:482] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:35.038501 29558 catalog_manager.cc:5484] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c reported cstate change: term changed from 4 to 5, leader changed from fdfeac4190e045f290e003b2298d988d (127.27.217.194) to 888d99522e4844118f8f1cf26f268c8c (127.27.217.196). New cstate: current_term: 5 leader_uuid: "888d99522e4844118f8f1cf26f268c8c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:01:35.032595 29658 raft_consensus.cc:1261] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Refusing update from remote peer 888d99522e4844118f8f1cf26f268c8c: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240501 20:01:35.094830 30965 consensus_queue.cc:260] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:35.097534 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 57114.6us
I20240501 20:01:35.101321 29700 leader_election.cc:302] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:35.098363 30966 raft_consensus.cc:3037] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:35.022028 30824 consensus_queue.cc:1035] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.056s
I20240501 20:01:35.091372 29735 rpcz_store.cc:276] 0501 20:01:33.580346 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:34.514248 (+933902us) service_pool.cc:225] Handling call
0501 20:01:35.059324 (+545076us) raft_consensus.cc:1457] Updating replica for 1 ops
0501 20:01:35.060429 (+ 1105us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:35.083621 (+ 23192us) spinlock_profiling.cc:243] Waited 425 ms on lock 0x7b54001077d0. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787ac59 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc 00007fe455e9e782 00007fe455ca2bcc 00007fe455ca2006 00007fe455ca19af
0501 20:01:35.087106 (+ 3485us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":977098624}
I20240501 20:01:35.084200 30650 raft_consensus.cc:686] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [term 2 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:35.090636 30753 raft_consensus.cc:729] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:35.035413 30852 consensus_queue.cc:1046] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.065s
I20240501 20:01:35.107282 30753 consensus_queue.cc:260] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:35.111338 30753 raft_consensus.cc:3037] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:35.111635 30852 raft_consensus.cc:3032] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:35.112124 30852 raft_consensus.cc:729] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:35.113240 30852 consensus_queue.cc:260] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:35.113613 30815 consensus_queue.cc:1046] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:01:35.108358 29809 raft_consensus.cc:2396] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240501 20:01:35.104359 30751 raft_consensus.cc:482] T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240501 20:01:35.107282 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 46807.6us
I20240501 20:01:35.108861 29730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 712) took 1541 ms. Trace:
I20240501 20:01:35.103264 30851 raft_consensus.cc:482] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:35.098991 30809 raft_consensus.cc:3037] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:35.114871 30894 raft_consensus.cc:2781] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:01:35.115375 29730 rpcz_store.cc:276] 0501 20:01:33.559546 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:34.494446 (+934900us) service_pool.cc:225] Handling call
0501 20:01:34.494783 (+ 337us) raft_consensus.cc:1457] Updating replica for 1 ops
0501 20:01:34.506952 (+ 12169us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:34.507023 (+ 71us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0501 20:01:34.507484 (+ 461us) log.cc:842] Serialized 26 byte log entry
0501 20:01:34.796513 (+289029us) raft_consensus.cc:1631] Marking committed up to 1
0501 20:01:34.796571 (+ 58us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:34.796706 (+ 135us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0501 20:01:35.034981 (+238275us) spinlock_profiling.cc:243] Waited 3.71 ms on lock 0x7b44000232cc. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe454797b51 00007fe454865685 00007fe4548660c2 00007fe45750e441 00007fe457509d4a 00007fe4574fdb1e 00007fe4574fce66 00007fe4574fca88 00007fe457506481 00007fe457506439 00007fe4575063c9
0501 20:01:35.077201 (+ 42220us) raft_consensus.cc:1671] finished
0501 20:01:35.077218 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:35.100690 (+ 23472us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12309,"raft.queue_time_us":10582,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":5430,"spinlock_wait_cycles":8529152,"thread_start_us":272100,"threads_started":1,"wal-append.queue_time_us":362184}
I20240501 20:01:35.116886 30849 raft_consensus.cc:3032] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:35.115443 29558 catalog_manager.cc:5484] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c reported cstate change: term changed from 2 to 3, leader changed from fdfeac4190e045f290e003b2298d988d (127.27.217.194) to 888d99522e4844118f8f1cf26f268c8c (127.27.217.196). New cstate: current_term: 3 leader_uuid: "888d99522e4844118f8f1cf26f268c8c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:01:35.117254 30751 raft_consensus.cc:504] T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:35.117589 29624 leader_election.cc:302] T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:35.120903 30851 raft_consensus.cc:504] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:35.104444 29805 raft_consensus.cc:1226] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 3. Current term is 4. Ops: []
I20240501 20:01:35.130363 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9046e38fc57f4364aa3d539e1fc01928" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:35.124361 30849 raft_consensus.cc:729] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:35.135169 30646 raft_consensus.cc:2726] T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 20:01:35.128209 30894 raft_consensus.cc:686] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:35.137147 30894 consensus_queue.cc:237] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:35.137993 29698 leader_election.cc:302] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:35.140905 30849 consensus_queue.cc:260] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:35.141767 29624 leader_election.cc:302] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:35.142237 30906 raft_consensus.cc:2781] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:01:35.142524 30849 raft_consensus.cc:3037] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:35.142858 30906 raft_consensus.cc:686] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [term 3 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:35.143627 30810 raft_consensus.cc:2726] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240501 20:01:35.145668 30965 raft_consensus.cc:3037] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:35.144337 30851 leader_election.cc:288] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:35.146644 30824 consensus_queue.cc:1035] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.006s
I20240501 20:01:35.145423 29624 leader_election.cc:302] T 2d86f8848a184a87947fd927c44e5a35 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:35.149132 29558 catalog_manager.cc:5484] T 7ed27033ab1645e9ab0600ab635d9589 P 888d99522e4844118f8f1cf26f268c8c reported cstate change: term changed from 2 to 4, leader changed from fdfeac4190e045f290e003b2298d988d (127.27.217.194) to 888d99522e4844118f8f1cf26f268c8c (127.27.217.196). New cstate: current_term: 4 leader_uuid: "888d99522e4844118f8f1cf26f268c8c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:01:35.151963 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9046e38fc57f4364aa3d539e1fc01928" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:35.173283 30852 raft_consensus.cc:3037] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:35.174126 30921 raft_consensus.cc:504] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:35.124516 30935 raft_consensus.cc:2781] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:01:35.176285 30921 leader_election.cc:288] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:35.180294 30650 consensus_queue.cc:237] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:35.181887 30713 raft_consensus.cc:2726] T 2d86f8848a184a87947fd927c44e5a35 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:35.182910 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d86f8848a184a87947fd927c44e5a35" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
W20240501 20:01:35.188665 31002 log.cc:924] Time spent T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.112s user 0.000s sys 0.001s
I20240501 20:01:35.189713 29800 raft_consensus.cc:1261] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Refusing update from remote peer 888d99522e4844118f8f1cf26f268c8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240501 20:01:35.190850 29804 raft_consensus.cc:3032] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:35.191020 29800 raft_consensus.cc:3032] T a8e9f75387564275a60d38e9a910319c P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:35.191249 29804 raft_consensus.cc:729] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:35.191377 29800 raft_consensus.cc:729] T a8e9f75387564275a60d38e9a910319c P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:35.193423 30933 consensus_queue.cc:1035] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:01:35.195088 29800 consensus_queue.cc:260] T a8e9f75387564275a60d38e9a910319c P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:35.197106 30384 leader_election.cc:302] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 888d99522e4844118f8f1cf26f268c8c; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:35.197544 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38cbeb0dc0c14e3d9182101eea885625" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:35.198778 30995 raft_consensus.cc:2726] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 20:01:35.196350 29800 raft_consensus.cc:3037] T a8e9f75387564275a60d38e9a910319c P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:35.143872 30906 consensus_queue.cc:237] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:35.212208 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec073fe645614fe28e7837de3c22510b" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:35.215386 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 75395.1us
I20240501 20:01:35.203294 30935 raft_consensus.cc:686] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 2 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:35.178786 30851 consensus_queue.cc:1046] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
W20240501 20:01:35.224772 30996 log.cc:924] Time spent T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d: Append to log took a long time: real 0.091s user 0.000s sys 0.000s
I20240501 20:01:35.226851 29808 raft_consensus.cc:3037] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:35.221405 29809 raft_consensus.cc:2379] T ec073fe645614fe28e7837de3c22510b P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 3: Already voted for candidate ee0d97247d394a8facfa774f15f3a67d in this term.
I20240501 20:01:35.219149 29804 consensus_queue.cc:260] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:35.276574 29625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 46537.5us
W20240501 20:01:35.281905 30998 log.cc:924] Time spent T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.110s user 0.000s sys 0.000s
I20240501 20:01:35.282272 30935 consensus_queue.cc:237] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:35.283067 30886 raft_consensus.cc:504] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:35.283957 29809 raft_consensus.cc:3032] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:01:35.284332 29809 raft_consensus.cc:729] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:35.285248 30886 leader_election.cc:288] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:35.285151 29809 consensus_queue.cc:260] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:35.286134 29809 raft_consensus.cc:3037] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:35.288446 29804 raft_consensus.cc:3037] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:01:35.288633 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 143322us
I20240501 20:01:35.289522 29624 leader_election.cc:302] T 010ceabf3e5a4b1c9330b80dade765bb P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:35.228674 30966 raft_consensus.cc:482] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:35.290441 30966 raft_consensus.cc:504] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:35.298552 29698 leader_election.cc:302] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:35.294159 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b337b8e3e38d4ad6b1076fbd0d0a1609" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:35.292614 30966 leader_election.cc:288] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:35.311800 29773 leader_election.cc:334] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 1.780s (SENT)
I20240501 20:01:35.280314 30713 raft_consensus.cc:482] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:35.315680 29773 leader_election.cc:302] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:35.311115 29802 raft_consensus.cc:1226] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20240501 20:01:35.321470 29800 raft_consensus.cc:1261] T a8e9f75387564275a60d38e9a910319c P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:01:35.326161 29804 raft_consensus.cc:1261] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:01:35.334154 30912 raft_consensus.cc:482] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:35.334861 30912 raft_consensus.cc:504] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:35.336177 30650 raft_consensus.cc:2781] T 010ceabf3e5a4b1c9330b80dade765bb P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:01:35.317386 30713 raft_consensus.cc:504] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:35.337867 30966 raft_consensus.cc:482] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:35.314851 30849 raft_consensus.cc:482] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:35.314056 30839 raft_consensus.cc:482] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:35.338160 30713 leader_election.cc:288] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:35.317778 30894 raft_consensus.cc:2726] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20240501 20:01:35.400358 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 61315.1us
I20240501 20:01:35.337900 30937 raft_consensus.cc:482] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
W20240501 20:01:35.360803 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 141314us
I20240501 20:01:35.327219 30751 leader_election.cc:288] T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:35.399377 29808 raft_consensus.cc:1261] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240501 20:01:35.331549 30809 raft_consensus.cc:504] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:35.435729 29624 leader_election.cc:302] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:35.417987 30650 raft_consensus.cc:686] T 010ceabf3e5a4b1c9330b80dade765bb P 4c8da1d580614449a636b855bf11b95f [term 2 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:35.443367 29801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e49dd5180f04313ab33942fb3ddc147" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:35.443660 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668b4a4d782c4278932dd8413dc3ab3e" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:35.444192 29801 raft_consensus.cc:2379] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 4: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:35.443380 30968 raft_consensus.cc:482] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:35.445993 30968 raft_consensus.cc:504] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:35.448011 30968 leader_election.cc:288] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:35.402181 30849 raft_consensus.cc:504] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:35.450830 29655 raft_consensus.cc:1226] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Rejecting Update request from peer 888d99522e4844118f8f1cf26f268c8c for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20240501 20:01:35.441548 30937 raft_consensus.cc:504] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:35.451588 30849 leader_election.cc:288] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:35.439033 30810 raft_consensus.cc:3032] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:01:35.452847 30810 raft_consensus.cc:729] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:35.453083 30937 leader_election.cc:288] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:35.452733 30650 consensus_queue.cc:237] T 010ceabf3e5a4b1c9330b80dade765bb P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:35.400326 30966 raft_consensus.cc:504] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:35.474182 29698 leader_election.cc:302] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
W20240501 20:01:35.477288 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 165509us
I20240501 20:01:35.453843 30810 consensus_queue.cc:260] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:35.483716 31008 log.cc:924] Time spent T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 0.183s user 0.002s sys 0.000s
I20240501 20:01:35.440215 30815 raft_consensus.cc:2726] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:35.488932 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfac893b4b4f457694a4eb184b6abb53" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:35.488955 29773 leader_election.cc:334] T ec073fe645614fe28e7837de3c22510b P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 1.859s (SENT)
I20240501 20:01:35.434794 30852 consensus_queue.cc:1035] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.045s
W20240501 20:01:35.493614 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 57973.6us
I20240501 20:01:35.494210 30935 raft_consensus.cc:2781] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:01:35.496829 30968 consensus_queue.cc:1035] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:35.497036 30713 consensus_queue.cc:1035] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:35.497660 29809 raft_consensus.cc:1261] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20240501 20:01:35.502882 29809 pending_rounds.cc:76] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d: Aborting all ops after (but not including) 1
I20240501 20:01:35.437103 30839 raft_consensus.cc:504] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } }
I20240501 20:01:35.503865 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e89c0572e4ee7be8e5315b459bdb3" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:35.503173 29809 pending_rounds.cc:97] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20240501 20:01:35.506350 29809 raft_consensus.cc:2866] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 20:01:35.487468 30886 raft_consensus.cc:2726] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:35.487496 30831 consensus_queue.cc:1035] T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20240501 20:01:35.510943 31017 raft_consensus.cc:482] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:35.511559 31017 raft_consensus.cc:504] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:35.513020 30966 leader_election.cc:288] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:35.513525 30809 leader_election.cc:288] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 election: Requested vote from peers ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:35.515192 30935 raft_consensus.cc:686] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:35.523741 30415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e89c0572e4ee7be8e5315b459bdb3" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c"
I20240501 20:01:35.524585 30912 leader_election.cc:288] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:35.525091 30944 consensus_queue.cc:1046] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:01:35.525946 30944 raft_consensus.cc:3032] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 4 LEADER]: Stepping down as leader of term 4
I20240501 20:01:35.526331 30944 raft_consensus.cc:729] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 888d99522e4844118f8f1cf26f268c8c, State: Running, Role: LEADER
I20240501 20:01:35.527158 30944 consensus_queue.cc:260] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:35.498653 31019 raft_consensus.cc:482] T 84fa28fb55f44b61ba1a88ed775a78ba P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:35.529096 30944 raft_consensus.cc:3037] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:35.528903 31019 raft_consensus.cc:504] T 84fa28fb55f44b61ba1a88ed775a78ba P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:35.527762 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e07220db7444ae97f294d70eda6a93" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:35.527316 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac2171683884f6f97cdfeb0a656cb49" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
W20240501 20:01:35.531368 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 89134.8us
I20240501 20:01:35.531771 29727 raft_consensus.cc:2379] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 3: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:35.532208 29698 leader_election.cc:302] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:35.525970 29623 leader_election.cc:302] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: 888d99522e4844118f8f1cf26f268c8c, ee0d97247d394a8facfa774f15f3a67d
W20240501 20:01:35.526906 29773 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 3.164s (SENT) [suppressed 61 similar messages]
I20240501 20:01:35.537442 29808 raft_consensus.cc:1261] T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20240501 20:01:35.538108 29808 pending_rounds.cc:76] T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d: Aborting all ops after (but not including) 1
I20240501 20:01:35.540444 29808 pending_rounds.cc:97] T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20240501 20:01:35.540710 29808 raft_consensus.cc:2866] T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 20:01:35.541074 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb27193abbf4644a48662909bc7b04a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:35.541702 29807 raft_consensus.cc:2454] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 2.
I20240501 20:01:35.531049 31019 leader_election.cc:288] T 84fa28fb55f44b61ba1a88ed775a78ba P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:35.542973 30937 raft_consensus.cc:2726] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240501 20:01:35.559712 31017 leader_election.cc:288] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:35.556174 30906 consensus_queue.cc:1035] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20240501 20:01:35.563300 30886 raft_consensus.cc:2726] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:35.548645 30912 raft_consensus.cc:482] T 3c5345f6eb62449ba69b4a9c2e349b31 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:35.570081 29735 raft_consensus.cc:1203] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Deduplicated request from leader. Original: 1.1->[3.2-3.2] Dedup: 3.2->[]
I20240501 20:01:35.570477 29735 raft_consensus.cc:1226] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20240501 20:01:35.571173 29806 raft_consensus.cc:1261] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20240501 20:01:35.576622 30894 consensus_queue.cc:1035] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 20:01:35.566938 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649b71f9f267437ca51c758dd9ca6987" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:35.577409 30935 consensus_queue.cc:237] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:35.568679 30912 raft_consensus.cc:504] T 3c5345f6eb62449ba69b4a9c2e349b31 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:35.581331 30810 raft_consensus.cc:3037] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:35.581342 30912 leader_election.cc:288] T 3c5345f6eb62449ba69b4a9c2e349b31 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:35.595773 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31fff64b453a423eb0f567b3171564fb" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:35.598306 29735 raft_consensus.cc:2354] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f for earlier term 3. Current term is 4.
I20240501 20:01:35.598392 30849 consensus_queue.cc:1046] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 20:01:35.603653 31015 log.cc:924] Time spent T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20240501 20:01:35.608879 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd5e83faaff4e2da4e4cc22ba410d33" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:35.616295 31021 raft_consensus.cc:482] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:35.617000 31021 raft_consensus.cc:504] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:35.619632 31021 leader_election.cc:288] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:35.621206 29698 leader_election.cc:302] T 010ceabf3e5a4b1c9330b80dade765bb P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:35.636593 29728 raft_consensus.cc:1203] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Deduplicated request from leader. Original: 1.1->[3.2-3.2] Dedup: 3.2->[]
I20240501 20:01:35.637243 29728 raft_consensus.cc:1226] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20240501 20:01:35.636612 29735 raft_consensus.cc:1226] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:35.660604 29809 raft_consensus.cc:3032] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:01:35.661036 29809 raft_consensus.cc:729] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:35.640726 29729 raft_consensus.cc:1203] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Deduplicated request from leader. Original: 1.1->[3.2-3.2] Dedup: 3.2->[]
I20240501 20:01:35.663976 29698 leader_election.cc:302] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:35.664417 29735 raft_consensus.cc:1226] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:35.667120 29809 consensus_queue.cc:260] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:35.668406 29809 raft_consensus.cc:3037] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:35.651137 30995 raft_consensus.cc:482] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:35.669843 30995 raft_consensus.cc:504] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:35.649005 30906 raft_consensus.cc:2726] T 010ceabf3e5a4b1c9330b80dade765bb P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:35.677201 29729 raft_consensus.cc:1226] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20240501 20:01:35.662772 30839 leader_election.cc:288] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:35.680691 30753 raft_consensus.cc:2726] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240501 20:01:35.680898 31024 ts_tablet_manager.cc:902] T 77be7101c0fe4085b6706296f609870a P 888d99522e4844118f8f1cf26f268c8c: Initiating tablet copy from peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:35.681437 30995 leader_election.cc:288] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:35.666563 30849 consensus_queue.cc:1046] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20240501 20:01:35.700210 29654 raft_consensus.cc:1226] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f [term 2 LEADER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:35.700670 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "456035a1cb3f4f7da52d0fc1dd1c656c" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:35.701102 29655 raft_consensus.cc:1226] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 2 LEADER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:35.701711 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec073fe645614fe28e7837de3c22510b" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:35.702061 29809 raft_consensus.cc:1261] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20240501 20:01:35.702777 29730 raft_consensus.cc:1226] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [term 2 LEADER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:35.703042 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f5d1686214485d91a0baf4b870740e" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:35.704109 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3f29f0eeea4b119984634202e5d9cd" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:35.704593 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458740ab5c444b6ab3d1991c69db5084" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:35.705324 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38cbeb0dc0c14e3d9182101eea885625" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:35.727528 29734 raft_consensus.cc:2379] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 3: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:35.705615 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c651044a305c4168805825a53d910899" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:35.706097 30769 raft_consensus.cc:482] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:35.706341 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec073fe645614fe28e7837de3c22510b" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:35.729800 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e49dd5180f04313ab33942fb3ddc147" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:35.730274 29726 raft_consensus.cc:2454] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 3.
I20240501 20:01:35.691710 30839 raft_consensus.cc:482] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:35.680013 29698 leader_election.cc:302] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:35.734331 29732 raft_consensus.cc:1261] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20240501 20:01:35.681862 30751 consensus_queue.cc:1046] T 48630ae6eb394ac1b54955dd01cf0ed2 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:35.701745 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31fff64b453a423eb0f567b3171564fb" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:35.726094 29801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "696121417ea9403990456cfc6fc4f416" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:35.719506 29809 pending_rounds.cc:76] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d: Aborting all ops after (but not including) 1
I20240501 20:01:35.728453 29735 raft_consensus.cc:2379] T c651044a305c4168805825a53d910899 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 2: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:35.734764 29732 pending_rounds.cc:76] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d: Aborting all ops after (but not including) 1
I20240501 20:01:35.742200 29732 pending_rounds.cc:97] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20240501 20:01:35.742645 29732 raft_consensus.cc:2866] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 20:01:35.743206 29809 pending_rounds.cc:97] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20240501 20:01:35.743649 29809 raft_consensus.cc:2866] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 20:01:35.745158 29726 raft_consensus.cc:1261] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240501 20:01:35.745390 29657 raft_consensus.cc:2379] T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 4: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:35.745077 29651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be383be783234855878cb8fd442c7b05" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:35.745972 29651 raft_consensus.cc:2454] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 2.
W20240501 20:01:35.746409 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 66543.3us
I20240501 20:01:35.747174 29731 raft_consensus.cc:1261] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240501 20:01:35.747444 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb27193abbf4644a48662909bc7b04a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:35.748075 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668b4a4d782c4278932dd8413dc3ab3e" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:35.745136 29730 raft_consensus.cc:1261] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:01:35.748584 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfac893b4b4f457694a4eb184b6abb53" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:35.749140 29652 raft_consensus.cc:3037] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:35.749325 29734 raft_consensus.cc:2379] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 3: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:35.750020 30753 raft_consensus.cc:2726] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 20:01:35.747179 29698 leader_election.cc:302] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:35.753446 31017 raft_consensus.cc:3032] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:35.753875 31017 raft_consensus.cc:729] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:35.754627 31017 consensus_queue.cc:260] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:35.727876 30965 raft_consensus.cc:482] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:35.755417 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69915dd5992641e3a24f7f45f1cd5c0c" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:35.755638 31017 raft_consensus.cc:3037] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:35.756248 29727 raft_consensus.cc:2379] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 3: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:35.756345 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb27193abbf4644a48662909bc7b04a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:35.729379 29729 raft_consensus.cc:2379] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 3: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:35.756718 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d2cac25f354a97b1bfb53d57ae7c08" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:35.684459 31016 consensus_queue.cc:1046] T 010ceabf3e5a4b1c9330b80dade765bb P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 20:01:35.757541 30886 raft_consensus.cc:2726] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240501 20:01:35.736757 30839 raft_consensus.cc:504] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:35.758430 29731 raft_consensus.cc:1261] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:01:35.759387 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af08384057a4d85977481472f460168" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:35.736205 30769 raft_consensus.cc:504] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:35.752660 29698 leader_election.cc:302] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:35.761709 30849 consensus_queue.cc:1046] T 36057b3f24d44b9aac3a1378053a395f P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 20:01:35.762163 30751 raft_consensus.cc:3032] T 48630ae6eb394ac1b54955dd01cf0ed2 P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:35.755924 30965 raft_consensus.cc:504] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:35.757058 29659 raft_consensus.cc:2454] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 2.
I20240501 20:01:35.723174 29658 raft_consensus.cc:3037] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:35.731873 29624 leader_election.cc:302] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: 888d99522e4844118f8f1cf26f268c8c, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:35.762594 30886 raft_consensus.cc:2726] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:01:35.776716 29698 leader_election.cc:302] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:35.806102 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "696121417ea9403990456cfc6fc4f416" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
W20240501 20:01:35.807735 29772 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 81584us
I20240501 20:01:35.762615 30751 raft_consensus.cc:729] T 48630ae6eb394ac1b54955dd01cf0ed2 P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:35.762221 30769 leader_election.cc:288] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:35.804956 30912 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
I20240501 20:01:35.776518 30972 raft_consensus.cc:3032] T 36057b3f24d44b9aac3a1378053a395f P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:35.814330 31021 raft_consensus.cc:2726] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:35.771790 30839 leader_election.cc:288] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:35.837209 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 115741us
I20240501 20:01:35.870990 29624 leader_election.cc:302] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:35.814021 30894 consensus_queue.cc:1035] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.051s
I20240501 20:01:35.829752 30972 raft_consensus.cc:729] T 36057b3f24d44b9aac3a1378053a395f P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
W20240501 20:01:35.886118 29772 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 54018us
I20240501 20:01:35.896981 30935 raft_consensus.cc:482] T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:35.883272 30886 raft_consensus.cc:482] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:35.873580 30384 leader_election.cc:302] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 888d99522e4844118f8f1cf26f268c8c; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:35.904327 31039 consensus_queue.cc:1046] T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20240501 20:01:35.900496 31029 log.cc:924] Time spent T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
W20240501 20:01:35.906971 29773 leader_election.cc:334] T 456035a1cb3f4f7da52d0fc1dd1c656c P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 2.182s (SENT)
I20240501 20:01:35.928889 29773 leader_election.cc:302] T 456035a1cb3f4f7da52d0fc1dd1c656c P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:35.908179 30935 raft_consensus.cc:504] T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:35.931217 30935 leader_election.cc:288] T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:35.925144 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 58706.5us
I20240501 20:01:35.926023 29652 raft_consensus.cc:2396] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240501 20:01:35.897189 31021 raft_consensus.cc:3037] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:35.937037 29624 leader_election.cc:302] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:35.942983 30849 raft_consensus.cc:2726] T 456035a1cb3f4f7da52d0fc1dd1c656c P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:01:35.927184 30906 raft_consensus.cc:2726] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:35.941617 30944 raft_consensus.cc:2726] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240501 20:01:35.951573 30384 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 113412us
W20240501 20:01:35.976580 31034 log.cc:924] Time spent T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.087s user 0.001s sys 0.000s
W20240501 20:01:35.963783 31033 log.cc:924] Time spent T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d: Append to log took a long time: real 0.053s user 0.000s sys 0.001s
W20240501 20:01:35.977551 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 70605.4us
I20240501 20:01:35.927541 30886 raft_consensus.cc:504] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:35.978989 29624 leader_election.cc:398] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Vote denied by peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907) with higher term. Message: Invalid argument: T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f for earlier term 3. Current term is 4.
I20240501 20:01:35.963346 30965 leader_election.cc:288] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:35.964404 30935 raft_consensus.cc:2726] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 20:01:35.932900 30751 consensus_queue.cc:260] T 48630ae6eb394ac1b54955dd01cf0ed2 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:36.011459 29624 leader_election.cc:401] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20240501 20:01:35.962963 30972 consensus_queue.cc:260] T 36057b3f24d44b9aac3a1378053a395f P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:35.977108 30769 raft_consensus.cc:482] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:36.017896 29658 raft_consensus.cc:2454] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 3.
W20240501 20:01:36.045814 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 67007.6us
I20240501 20:01:36.046691 29624 leader_election.cc:302] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: 888d99522e4844118f8f1cf26f268c8c, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:36.048112 30851 raft_consensus.cc:2726] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20240501 20:01:36.052695 31038 log.cc:924] Time spent T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.095s user 0.001s sys 0.000s
I20240501 20:01:36.053442 30886 leader_election.cc:288] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:36.055382 29773 leader_election.cc:334] T a8787bc4d6b44649b9138d906730110a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 2.240s (SENT)
I20240501 20:01:36.055610 30965 consensus_queue.cc:1046] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:36.055850 29773 leader_election.cc:302] T a8787bc4d6b44649b9138d906730110a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:36.022624 30935 raft_consensus.cc:2726] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907) with higher term. Message: Invalid argument: T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f for earlier term 3. Current term is 4.
I20240501 20:01:36.056344 30965 consensus_queue.cc:1225] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer fdfeac4190e045f290e003b2298d988d log is divergent from this leader: its last log entry 4.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20240501 20:01:36.057614 30769 raft_consensus.cc:504] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:36.059652 31019 raft_consensus.cc:2726] T a8787bc4d6b44649b9138d906730110a P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:36.066617 31020 raft_consensus.cc:3032] T 010ceabf3e5a4b1c9330b80dade765bb P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:36.067070 31020 raft_consensus.cc:729] T 010ceabf3e5a4b1c9330b80dade765bb P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:36.059731 30769 leader_election.cc:288] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:36.075143 31020 consensus_queue.cc:260] T 010ceabf3e5a4b1c9330b80dade765bb P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:36.076661 29624 leader_election.cc:302] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:36.076892 31020 raft_consensus.cc:3037] T 010ceabf3e5a4b1c9330b80dade765bb P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:36.078861 30851 raft_consensus.cc:2726] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:01:36.082746 30815 raft_consensus.cc:3032] T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:01:36.083289 30815 raft_consensus.cc:729] T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:36.084352 30815 consensus_queue.cc:260] T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:36.085883 29624 leader_election.cc:302] T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:36.086899 30769 raft_consensus.cc:3037] T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 3
W20240501 20:01:36.097307 31032 log.cc:924] Time spent T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 0.270s user 0.001s sys 0.002s
I20240501 20:01:36.099524 30751 raft_consensus.cc:3037] T 48630ae6eb394ac1b54955dd01cf0ed2 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:36.099952 30972 raft_consensus.cc:3037] T 36057b3f24d44b9aac3a1378053a395f P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:36.088779 29624 leader_election.cc:302] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:36.106374 29624 leader_election.cc:302] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:36.114718 30851 raft_consensus.cc:3037] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:36.107678 30650 raft_consensus.cc:2741] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20240501 20:01:36.122797 29624 leader_election.cc:302] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
W20240501 20:01:36.129527 29772 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 242669us
I20240501 20:01:36.132658 29801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcf30cf66714efdac997862dafbe4a2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:36.134816 29624 leader_election.cc:302] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:36.133352 29801 raft_consensus.cc:2454] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 3.
I20240501 20:01:36.140003 29625 leader_election.cc:302] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters: fdfeac4190e045f290e003b2298d988d
I20240501 20:01:36.141539 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af08384057a4d85977481472f460168" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:36.144467 29624 leader_election.cc:302] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:36.131803 30935 raft_consensus.cc:3032] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:01:36.132220 30839 raft_consensus.cc:2726] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 20:01:36.150209 29808 raft_consensus.cc:1226] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20240501 20:01:36.150296 29804 raft_consensus.cc:1261] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240501 20:01:36.139030 29700 leader_election.cc:302] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:36.155690 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69915dd5992641e3a24f7f45f1cd5c0c" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:36.140787 30886 raft_consensus.cc:482] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:36.160162 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d2cac25f354a97b1bfb53d57ae7c08" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:36.143775 31017 raft_consensus.cc:482] T d1d2cac25f354a97b1bfb53d57ae7c08 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:36.162684 30414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05df43bdfa479b902b1da297fb5974" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c" is_pre_election: true
I20240501 20:01:36.152829 30808 consensus_queue.cc:1035] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20240501 20:01:36.149883 30646 raft_consensus.cc:2726] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:36.136245 30965 consensus_queue.cc:1035] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20240501 20:01:36.141544 30769 raft_consensus.cc:2726] T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:01:36.152262 30815 raft_consensus.cc:3037] T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:36.136152 30650 consensus_queue.cc:1035] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
I20240501 20:01:36.168794 30414 raft_consensus.cc:2454] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 3.
I20240501 20:01:36.147575 30935 raft_consensus.cc:729] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:36.160989 30839 raft_consensus.cc:2781] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:36.156746 30851 raft_consensus.cc:2726] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240501 20:01:36.194955 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 55995.3us
I20240501 20:01:36.167843 30810 raft_consensus.cc:3037] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 3
W20240501 20:01:36.200721 29773 leader_election.cc:334] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 1.710s (SENT)
I20240501 20:01:36.201279 29773 leader_election.cc:302] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:36.197593 31040 raft_consensus.cc:2781] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:36.196899 30839 raft_consensus.cc:482] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:36.168089 30886 raft_consensus.cc:504] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:36.181792 31017 raft_consensus.cc:504] T d1d2cac25f354a97b1bfb53d57ae7c08 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:36.218286 31040 raft_consensus.cc:482] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:36.214783 31019 raft_consensus.cc:2726] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:36.223274 30839 raft_consensus.cc:3037] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:36.234346 29804 raft_consensus.cc:2454] T d1d2cac25f354a97b1bfb53d57ae7c08 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 2.
I20240501 20:01:36.240079 29801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b337b8e3e38d4ad6b1076fbd0d0a1609" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:36.229079 30965 raft_consensus.cc:482] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:36.247239 30965 raft_consensus.cc:504] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:36.249679 30965 leader_election.cc:288] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:36.231967 31040 raft_consensus.cc:3037] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:36.235231 30769 raft_consensus.cc:482] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:36.223434 30935 consensus_queue.cc:260] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:36.260958 29773 leader_election.cc:334] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 2.141s (SENT)
I20240501 20:01:36.266301 29801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8787bc4d6b44649b9138d906730110a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:36.267109 29801 raft_consensus.cc:2454] T a8787bc4d6b44649b9138d906730110a P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 1.
I20240501 20:01:36.271365 29773 leader_election.cc:302] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:36.261430 30769 raft_consensus.cc:504] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:36.307595 29801 raft_consensus.cc:3032] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:36.308171 29801 raft_consensus.cc:729] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:36.312227 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e38a2e91a3d48d09a22abce723af833" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:36.293877 30935 raft_consensus.cc:3037] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:36.281183 31046 raft_consensus.cc:2726] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:36.296422 29802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d88139f89c47ff84cad674147aae0f" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:36.321933 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 64783.2us
W20240501 20:01:36.322515 29625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 182573us
I20240501 20:01:36.325930 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36057b3f24d44b9aac3a1378053a395f" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:36.319535 29801 consensus_queue.cc:260] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:36.329118 29801 raft_consensus.cc:3037] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:36.330159 29772 leader_election.cc:302] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 888d99522e4844118f8f1cf26f268c8c, ee0d97247d394a8facfa774f15f3a67d; no voters:
W20240501 20:01:36.333232 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 188869us
I20240501 20:01:36.335950 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31fff64b453a423eb0f567b3171564fb" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:36.340571 29803 raft_consensus.cc:2454] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 4.
W20240501 20:01:36.335837 29698 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.506s user 0.034s sys 0.038s
I20240501 20:01:36.337776 30650 consensus_queue.cc:1035] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
W20240501 20:01:36.341899 29698 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.512s user 0.034s sys 0.038s
W20240501 20:01:36.336527 30384 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 170466us
I20240501 20:01:36.342378 29700 leader_election.cc:302] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
W20240501 20:01:36.342520 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 565851us
I20240501 20:01:36.343289 31042 raft_consensus.cc:2781] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240501 20:01:36.343662 31042 raft_consensus.cc:482] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:36.343994 31042 raft_consensus.cc:3037] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:36.334256 30886 leader_election.cc:288] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:36.344936 29624 leader_election.cc:302] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:36.332576 31017 leader_election.cc:288] T d1d2cac25f354a97b1bfb53d57ae7c08 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:36.331511 30851 consensus_queue.cc:1035] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20240501 20:01:36.347885 29625 leader_election.cc:302] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:36.359108 29624 leader_election.cc:302] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:36.372619 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece8d4ed84c34b748f7b6272b31c90ac" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:36.351363 30815 raft_consensus.cc:2781] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:36.375473 29625 leader_election.cc:302] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters: fdfeac4190e045f290e003b2298d988d
I20240501 20:01:36.357249 30646 raft_consensus.cc:3037] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:36.378906 31052 raft_consensus.cc:2781] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:36.379465 31052 raft_consensus.cc:482] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:36.379866 31052 raft_consensus.cc:3037] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:36.377995 30965 raft_consensus.cc:2726] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240501 20:01:36.375041 30815 raft_consensus.cc:482] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:36.351696 31043 consensus_queue.cc:1035] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20240501 20:01:36.390200 30808 consensus_queue.cc:1035] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
W20240501 20:01:36.404285 31049 log.cc:924] Time spent T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d: Append to log took a long time: real 0.066s user 0.000s sys 0.001s
I20240501 20:01:36.388938 30650 raft_consensus.cc:2781] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:36.411180 30650 raft_consensus.cc:482] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:36.411679 30650 raft_consensus.cc:3037] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:36.393250 31021 raft_consensus.cc:2726] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:01:36.394716 30815 raft_consensus.cc:3037] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:36.420634 29801 raft_consensus.cc:1261] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 2. (term mismatch)
I20240501 20:01:36.421228 29801 pending_rounds.cc:76] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d: Aborting all ops after (but not including) 1
I20240501 20:01:36.421581 29801 pending_rounds.cc:97] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 1.2
I20240501 20:01:36.424741 29801 raft_consensus.cc:2894] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Aborting config change with OpId 1.2: Aborted: Op aborted by new leader
I20240501 20:01:36.425364 29801 raft_consensus.cc:2866] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20240501 20:01:36.421775 30886 raft_consensus.cc:482] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:36.434162 30886 raft_consensus.cc:504] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:36.436237 30886 leader_election.cc:288] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:36.438267 29804 raft_consensus.cc:2454] T 36057b3f24d44b9aac3a1378053a395f P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 2.
I20240501 20:01:36.437968 31042 raft_consensus.cc:504] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:36.439262 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8787bc4d6b44649b9138d906730110a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:36.439949 29805 raft_consensus.cc:2454] T a8787bc4d6b44649b9138d906730110a P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
I20240501 20:01:36.420571 30769 leader_election.cc:288] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:36.440855 29625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 64188.7us
W20240501 20:01:36.442426 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 68013.6us
I20240501 20:01:36.443501 29808 raft_consensus.cc:1261] T 010ceabf3e5a4b1c9330b80dade765bb P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:01:36.443722 29801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:49458 (request call id 3) took 3875 ms. Trace:
I20240501 20:01:36.453787 31024 tablet_copy_client.cc:308] T 77be7101c0fe4085b6706296f609870a P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Beginning tablet copy session from remote peer at address 127.27.217.195:35861
I20240501 20:01:36.443867 29625 leader_election.cc:302] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:36.454876 31042 leader_election.cc:288] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 5 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:36.444954 29801 rpcz_store.cc:276] 0501 20:01:32.566901 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:33.822348 (+1255447us) service_pool.cc:225] Handling call
0501 20:01:33.874402 (+ 52054us) log.cc:842] Serialized 513 byte log entry
0501 20:01:36.440423 (+2566021us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":115920,"raft.queue_time_us":300146,"raft.run_cpu_time_us":2275,"raft.run_wall_time_us":132033,"spinlock_wait_cycles":1968896,"thread_start_us":558609,"threads_started":2,"wal-append.queue_time_us":256300,"wal-append.run_cpu_time_us":2553,"wal-append.run_wall_time_us":1141183}
I20240501 20:01:36.444660 29700 leader_election.cc:302] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:36.457998 30851 raft_consensus.cc:2781] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:36.458495 30851 raft_consensus.cc:482] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:36.445286 29624 leader_election.cc:302] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:36.458843 30851 raft_consensus.cc:3037] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:36.459954 29819 tablet_copy_service.cc:140] P ee0d97247d394a8facfa774f15f3a67d: Received BeginTabletCopySession request for tablet 77be7101c0fe4085b6706296f609870a from peer 888d99522e4844118f8f1cf26f268c8c ({username='slave'} at 127.0.0.1:49474)
I20240501 20:01:36.460680 29819 tablet_copy_service.cc:161] P ee0d97247d394a8facfa774f15f3a67d: Beginning new tablet copy session on tablet 77be7101c0fe4085b6706296f609870a from peer 888d99522e4844118f8f1cf26f268c8c at {username='slave'} at 127.0.0.1:49474: session id = 888d99522e4844118f8f1cf26f268c8c-77be7101c0fe4085b6706296f609870a
I20240501 20:01:36.465569 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31fff64b453a423eb0f567b3171564fb" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:36.466230 29805 raft_consensus.cc:3037] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:36.467869 30851 raft_consensus.cc:504] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:36.470731 30769 raft_consensus.cc:2726] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:36.472265 31050 raft_consensus.cc:2781] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:36.473232 31050 raft_consensus.cc:482] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:36.473567 31050 raft_consensus.cc:3037] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:36.478070 30414 raft_consensus.cc:1261] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20240501 20:01:36.478497 30414 pending_rounds.cc:76] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c: Aborting all ops after (but not including) 4
I20240501 20:01:36.478804 30414 pending_rounds.cc:97] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20240501 20:01:36.479094 30414 raft_consensus.cc:2866] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 20:01:36.480214 30972 consensus_queue.cc:1046] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:36.481076 31060 raft_consensus.cc:3032] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:01:36.481449 31060 raft_consensus.cc:729] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:36.482342 31060 consensus_queue.cc:260] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:36.483683 31060 raft_consensus.cc:3037] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:36.471966 30646 raft_consensus.cc:2726] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:01:36.483817 31017 consensus_queue.cc:1046] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 20:01:36.485049 31017 consensus_queue.cc:1046] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:36.486147 31046 raft_consensus.cc:3032] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:36.486475 31046 raft_consensus.cc:729] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:36.487265 31046 consensus_queue.cc:260] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:36.488467 31046 raft_consensus.cc:3037] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:36.488500 30849 raft_consensus.cc:3032] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:01:36.489012 30849 raft_consensus.cc:729] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:36.489775 30849 consensus_queue.cc:260] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } }
I20240501 20:01:36.490669 29658 raft_consensus.cc:1261] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240501 20:01:36.491231 29805 raft_consensus.cc:2454] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 5.
I20240501 20:01:36.491827 30886 raft_consensus.cc:482] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:36.494088 29652 raft_consensus.cc:1226] T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 3. Current term is 4. Ops: []
I20240501 20:01:36.492149 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9472c53f79ec423f82c396bd980674e2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:36.492558 29700 leader_election.cc:302] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:36.499624 30849 raft_consensus.cc:3037] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:36.500149 30753 raft_consensus.cc:2781] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:01:36.500687 30753 raft_consensus.cc:686] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [term 5 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:36.500793 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49da4246f5f6438abe0e7e8c9a523d80" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:36.501375 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69915dd5992641e3a24f7f45f1cd5c0c" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:36.501613 29656 raft_consensus.cc:2396] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240501 20:01:36.502554 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c651044a305c4168805825a53d910899" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:36.503166 29651 raft_consensus.cc:1261] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20240501 20:01:36.503594 29651 pending_rounds.cc:76] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f: Aborting all ops after (but not including) 1
I20240501 20:01:36.503856 29651 pending_rounds.cc:97] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20240501 20:01:36.504201 29651 raft_consensus.cc:2866] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 20:01:36.504930 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e49dd5180f04313ab33942fb3ddc147" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:36.505739 29655 raft_consensus.cc:2396] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240501 20:01:36.506183 29653 raft_consensus.cc:1261] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20240501 20:01:36.506532 29653 pending_rounds.cc:76] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f: Aborting all ops after (but not including) 1
I20240501 20:01:36.506494 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5525f546426c47a38edf7f56ad04b7ab" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:36.507014 29653 pending_rounds.cc:97] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20240501 20:01:36.493723 30886 raft_consensus.cc:504] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:36.503696 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:36.507514 29655 raft_consensus.cc:2454] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 3.
I20240501 20:01:36.508698 29659 raft_consensus.cc:1261] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20240501 20:01:36.500010 29652 raft_consensus.cc:1226] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 2. Current term is 3. Ops: []
I20240501 20:01:36.503271 29660 raft_consensus.cc:2379] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d in current term 2: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:36.509833 30886 leader_election.cc:288] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:36.480556 31020 consensus_queue.cc:1046] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.027s
I20240501 20:01:36.510785 30972 raft_consensus.cc:3032] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:01:36.507316 29653 raft_consensus.cc:2866] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 20:01:36.511140 31043 raft_consensus.cc:482] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:36.511327 30972 raft_consensus.cc:729] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:36.505226 29651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38cbeb0dc0c14e3d9182101eea885625" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:36.511731 31043 raft_consensus.cc:504] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:36.512867 29651 raft_consensus.cc:2396] T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240501 20:01:36.512472 30972 consensus_queue.cc:260] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:36.513898 31043 leader_election.cc:288] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:36.514284 30972 raft_consensus.cc:3037] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:36.514588 29652 raft_consensus.cc:1261] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20240501 20:01:36.515007 29652 pending_rounds.cc:76] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f: Aborting all ops after (but not including) 1
I20240501 20:01:36.515244 29652 pending_rounds.cc:97] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20240501 20:01:36.515440 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e38a2e91a3d48d09a22abce723af833" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:36.515621 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211e27a3b0084b89a6dd7550849b5a65" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:36.516034 29660 raft_consensus.cc:3037] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:36.516892 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b337b8e3e38d4ad6b1076fbd0d0a1609" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:36.517477 29655 raft_consensus.cc:3037] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:36.514447 31043 raft_consensus.cc:482] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:36.518877 31043 raft_consensus.cc:504] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:36.520783 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d86f8848a184a87947fd927c44e5a35" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
W20240501 20:01:36.521363 29606 auto_rebalancer.cc:254] failed to send replica move request: Aborted: Op aborted by new leader
I20240501 20:01:36.521723 31043 leader_election.cc:288] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:36.521880 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49da4246f5f6438abe0e7e8c9a523d80" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:36.522851 29653 raft_consensus.cc:2396] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240501 20:01:36.523515 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1a08b455b94ae089396a15c837ddc2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:36.524125 29806 raft_consensus.cc:2454] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 2.
I20240501 20:01:36.515463 29652 raft_consensus.cc:2866] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 20:01:36.511003 29656 raft_consensus.cc:1261] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20240501 20:01:36.501518 30753 consensus_queue.cc:237] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:36.521390 29659 raft_consensus.cc:3037] T 2d86f8848a184a87947fd927c44e5a35 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:36.480146 30810 raft_consensus.cc:2726] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:01:36.527400 31017 raft_consensus.cc:482] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:36.527901 31017 raft_consensus.cc:504] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:36.529866 31017 leader_election.cc:288] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
W20240501 20:01:36.528270 29606 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 77be7101c0fe4085b6706296f609870a, TS fdfeac4190e045f290e003b2298d988d -> TS 888d99522e4844118f8f1cf26f268c8c move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:36.530447 29606 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 77be7101c0fe4085b6706296f609870a, TS fdfeac4190e045f290e003b2298d988d -> TS 888d99522e4844118f8f1cf26f268c8c move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:36.532480 29772 leader_election.cc:334] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959): Timed out: RequestConsensusVote RPC to 127.27.217.193:33959 timed out after 1.524s (SENT)
I20240501 20:01:36.533013 29772 leader_election.cc:302] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:36.525667 29656 pending_rounds.cc:76] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f: Aborting all ops after (but not including) 1
I20240501 20:01:36.534479 30751 raft_consensus.cc:2726] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:36.534919 29656 pending_rounds.cc:97] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20240501 20:01:36.535176 30751 raft_consensus.cc:482] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:36.535229 29656 raft_consensus.cc:2866] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 20:01:36.535785 30751 raft_consensus.cc:504] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:36.526201 29700 leader_election.cc:302] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:36.537598 31045 raft_consensus.cc:2781] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:36.538651 31045 raft_consensus.cc:482] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240501 20:01:36.538820 29606 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 7a3f29f0eeea4b119984634202e5d9cd, TS 4c8da1d580614449a636b855bf11b95f -> TS 888d99522e4844118f8f1cf26f268c8c move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:36.539031 31045 raft_consensus.cc:3037] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Advancing to term 3
W20240501 20:01:36.539175 29606 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 7a3f29f0eeea4b119984634202e5d9cd, TS 4c8da1d580614449a636b855bf11b95f -> TS 888d99522e4844118f8f1cf26f268c8c move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:36.541680 29651 raft_consensus.cc:1261] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240501 20:01:36.537688 30751 leader_election.cc:288] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:36.544992 31042 raft_consensus.cc:482] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:36.545480 31042 raft_consensus.cc:504] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:36.546027 29606 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 0fb27193abbf4644a48662909bc7b04a, TS ee0d97247d394a8facfa774f15f3a67d -> TS 888d99522e4844118f8f1cf26f268c8c move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:36.522061 29657 raft_consensus.cc:2454] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 3.
I20240501 20:01:36.495486 31040 raft_consensus.cc:504] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:36.548986 29804 raft_consensus.cc:1261] T d1d2cac25f354a97b1bfb53d57ae7c08 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:01:36.549440 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77be7101c0fe4085b6706296f609870a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:36.550487 29800 raft_consensus.cc:2454] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 2.
I20240501 20:01:36.550639 31040 leader_election.cc:288] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:36.550712 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c597f15e844f30837012baed4b0501" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:36.549193 29606 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 0fb27193abbf4644a48662909bc7b04a, TS ee0d97247d394a8facfa774f15f3a67d -> TS 888d99522e4844118f8f1cf26f268c8c move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:36.548255 31042 leader_election.cc:288] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:36.553866 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcf30cf66714efdac997862dafbe4a2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:36.554729 29700 leader_election.cc:302] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:36.544104 30839 raft_consensus.cc:504] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:36.562716 29660 raft_consensus.cc:2396] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240501 20:01:36.537979 30815 raft_consensus.cc:504] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:36.565322 29772 leader_election.cc:302] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:36.567404 29660 raft_consensus.cc:1261] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
W20240501 20:01:36.555678 29698 connection.cc:232] RPC call timeout handler was delayed by -1.0233337690001463s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1705s for a response
W20240501 20:01:36.570794 29606 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet a8e9f75387564275a60d38e9a910319c, TS fdfeac4190e045f290e003b2298d988d -> TS 888d99522e4844118f8f1cf26f268c8c move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:36.570904 29698 leader_election.cc:302] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, 888d99522e4844118f8f1cf26f268c8c
W20240501 20:01:36.571211 29606 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet a8e9f75387564275a60d38e9a910319c, TS fdfeac4190e045f290e003b2298d988d -> TS 888d99522e4844118f8f1cf26f268c8c move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:36.531808 29658 raft_consensus.cc:2454] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 2.
I20240501 20:01:36.574204 29657 raft_consensus.cc:1261] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20240501 20:01:36.573127 29660 pending_rounds.cc:76] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f: Aborting all ops after (but not including) 1
W20240501 20:01:36.577847 29698 consensus_peers.cc:485] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d -> Peer 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227): Couldn't send request to peer 888d99522e4844118f8f1cf26f268c8c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:36.577546 29660 pending_rounds.cc:97] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20240501 20:01:36.568956 31017 raft_consensus.cc:2781] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:36.571120 31042 raft_consensus.cc:2781] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:36.588408 29659 raft_consensus.cc:2454] T 2d86f8848a184a87947fd927c44e5a35 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 3.
I20240501 20:01:36.577495 29657 pending_rounds.cc:76] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f: Aborting all ops after (but not including) 1
I20240501 20:01:36.591605 29657 pending_rounds.cc:97] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20240501 20:01:36.574021 31044 raft_consensus.cc:2726] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:36.566489 31020 consensus_queue.cc:1046] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20240501 20:01:36.582202 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8787bc4d6b44649b9138d906730110a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:36.580138 29660 raft_consensus.cc:2866] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 20:01:36.559826 30935 consensus_queue.cc:1035] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.015s
I20240501 20:01:36.583206 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b337b8e3e38d4ad6b1076fbd0d0a1609" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:36.563112 31043 consensus_queue.cc:1035] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20240501 20:01:36.593300 29772 leader_election.cc:302] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:36.594210 29809 raft_consensus.cc:1226] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 3. Current term is 4. Ops: []
I20240501 20:01:36.589787 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d88139f89c47ff84cad674147aae0f" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:36.592000 29657 raft_consensus.cc:2866] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 20:01:36.598798 29733 raft_consensus.cc:2454] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 2.
I20240501 20:01:36.599206 29730 raft_consensus.cc:2379] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 3: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:36.575980 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458740ab5c444b6ab3d1991c69db5084" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:36.596665 30751 raft_consensus.cc:482] T f6d88139f89c47ff84cad674147aae0f P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:36.610975 29726 raft_consensus.cc:1261] T 010ceabf3e5a4b1c9330b80dade765bb P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:01:36.591059 31017 raft_consensus.cc:482] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:36.613934 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36057b3f24d44b9aac3a1378053a395f" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:36.614688 29730 raft_consensus.cc:2454] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 2.
I20240501 20:01:36.616925 31017 raft_consensus.cc:3037] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:36.610195 31044 consensus_queue.cc:1046] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
I20240501 20:01:36.618165 31044 raft_consensus.cc:3032] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:01:36.618501 31044 raft_consensus.cc:729] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:36.619230 31044 consensus_queue.cc:260] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:36.620450 30839 leader_election.cc:288] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:36.623569 31052 raft_consensus.cc:504] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:36.624332 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcf30cf66714efdac997862dafbe4a2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:36.625762 29819 tablet_copy_source_session.cc:205] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:01:36.625842 31052 leader_election.cc:288] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:36.627717 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e38a2e91a3d48d09a22abce723af833" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:36.628727 29727 raft_consensus.cc:2379] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 3: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:36.589563 30894 consensus_queue.cc:1035] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
I20240501 20:01:36.631768 30650 raft_consensus.cc:504] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:36.632313 29656 raft_consensus.cc:1226] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 2. Current term is 3. Ops: []
I20240501 20:01:36.633136 30912 raft_consensus.cc:3032] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:01:36.633498 30912 raft_consensus.cc:729] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:36.633824 30650 leader_election.cc:288] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:36.634263 30912 consensus_queue.cc:260] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:36.634927 29651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8787bc4d6b44649b9138d906730110a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:36.635226 30912 raft_consensus.cc:3037] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:36.635748 29651 raft_consensus.cc:2379] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d in current term 3: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:36.616271 31020 raft_consensus.cc:2781] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:36.636721 31020 raft_consensus.cc:482] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:36.616936 30751 raft_consensus.cc:504] T f6d88139f89c47ff84cad674147aae0f P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:36.637166 31020 raft_consensus.cc:3037] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:36.637573 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31fff64b453a423eb0f567b3171564fb" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:36.638157 29654 raft_consensus.cc:1226] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 2. Current term is 3. Ops: []
I20240501 20:01:36.638100 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8cd40cb4044d18a86001ce6ddf97ab" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:36.638635 30751 leader_election.cc:288] T f6d88139f89c47ff84cad674147aae0f P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:36.591348 31042 raft_consensus.cc:482] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:36.644125 29655 raft_consensus.cc:2454] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 3.
I20240501 20:01:36.585686 30851 leader_election.cc:288] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:36.638134 29653 raft_consensus.cc:3037] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:36.636968 29652 raft_consensus.cc:1261] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240501 20:01:36.628250 31050 raft_consensus.cc:504] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:36.641844 29773 leader_election.cc:334] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 1.905s (SENT)
I20240501 20:01:36.658926 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece8d4ed84c34b748f7b6272b31c90ac" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:36.659359 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c597f15e844f30837012baed4b0501" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:36.645645 31042 raft_consensus.cc:3037] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:36.547561 31045 raft_consensus.cc:504] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:36.666543 29731 raft_consensus.cc:2379] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 3: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:36.653999 30849 raft_consensus.cc:482] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:36.654412 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "696121417ea9403990456cfc6fc4f416" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
W20240501 20:01:36.671945 29773 leader_election.cc:334] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 1.931s (SENT)
I20240501 20:01:36.691937 31060 raft_consensus.cc:482] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:36.662107 31044 raft_consensus.cc:3037] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:36.697434 29773 leader_election.cc:302] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:36.671672 30839 consensus_queue.cc:1035] T 010ceabf3e5a4b1c9330b80dade765bb P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
I20240501 20:01:36.705391 30972 raft_consensus.cc:2726] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:36.706790 30646 consensus_queue.cc:1035] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:36.666460 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31fff64b453a423eb0f567b3171564fb" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:36.708472 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84fa28fb55f44b61ba1a88ed775a78ba" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:36.709321 31050 leader_election.cc:288] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:36.710070 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05df43bdfa479b902b1da297fb5974" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:36.711191 31045 leader_election.cc:288] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:36.706445 31060 raft_consensus.cc:504] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:36.711251 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec073fe645614fe28e7837de3c22510b" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:36.712659 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a2bba62f434d1c8489b55c8fec2187" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:36.713207 31060 leader_election.cc:288] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:36.713853 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "696121417ea9403990456cfc6fc4f416" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:36.715406 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec073fe645614fe28e7837de3c22510b" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:36.724925 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05df43bdfa479b902b1da297fb5974" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:36.726411 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5525fc4b82a44b0991c3dd6098fde81" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:36.727627 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31fff64b453a423eb0f567b3171564fb" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:36.730580 30646 raft_consensus.cc:482] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:36.742216 30646 raft_consensus.cc:504] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:36.742852 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f261d2a05bf4c838532049601694b4a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:36.744074 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5525f546426c47a38edf7f56ad04b7ab" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:36.744284 30646 leader_election.cc:288] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:36.744675 29735 raft_consensus.cc:2454] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 3.
I20240501 20:01:36.711668 30815 leader_election.cc:288] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:36.736030 31024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77be7101c0fe4085b6706296f609870a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:36.745479 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d88139f89c47ff84cad674147aae0f" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:36.746160 29735 raft_consensus.cc:2379] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 2: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:36.737531 30415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05df43bdfa479b902b1da297fb5974" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c"
I20240501 20:01:36.737066 30414 raft_consensus.cc:1261] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240501 20:01:36.748351 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb27193abbf4644a48662909bc7b04a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:36.750320 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1a08b455b94ae089396a15c837ddc2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:36.751314 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77be7101c0fe4085b6706296f609870a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:36.753459 29726 raft_consensus.cc:2454] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 3.
I20240501 20:01:36.764853 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36057b3f24d44b9aac3a1378053a395f" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:36.778656 29734 raft_consensus.cc:3037] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:36.779448 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be383be783234855878cb8fd442c7b05" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:36.779392 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d2cac25f354a97b1bfb53d57ae7c08" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:36.781035 29558 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33356 (request call id 50) took 1847 ms. Trace:
I20240501 20:01:36.788540 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458740ab5c444b6ab3d1991c69db5084" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:36.788918 29728 raft_consensus.cc:1226] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 1. Current term is 3. Ops: []
I20240501 20:01:36.789781 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9472c53f79ec423f82c396bd980674e2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:36.790060 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05df43bdfa479b902b1da297fb5974" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:36.790501 29729 raft_consensus.cc:2454] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 2.
I20240501 20:01:36.791254 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:36.791420 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5345f6eb62449ba69b4a9c2e349b31" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:36.792479 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36057b3f24d44b9aac3a1378053a395f" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:36.793025 29726 raft_consensus.cc:1261] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240501 20:01:36.795785 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8787bc4d6b44649b9138d906730110a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:36.793067 29805 raft_consensus.cc:3037] T 36057b3f24d44b9aac3a1378053a395f P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:36.796355 29807 raft_consensus.cc:3037] T a8787bc4d6b44649b9138d906730110a P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 3
I20240501 20:01:36.789798 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d2cac25f354a97b1bfb53d57ae7c08" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:36.779877 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb27193abbf4644a48662909bc7b04a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:36.788635 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07bdb8b09b8c4ef4b8e49daa9b8c01d6" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:36.797830 29652 raft_consensus.cc:1226] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 2. Current term is 3. Ops: []
I20240501 20:01:36.787636 29654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211e27a3b0084b89a6dd7550849b5a65" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:36.802214 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8787bc4d6b44649b9138d906730110a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:36.802865 29735 raft_consensus.cc:1226] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20240501 20:01:36.781534 29558 rpcz_store.cc:276] 0501 20:01:34.917431 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:34.917674 (+ 243us) service_pool.cc:225] Handling call
0501 20:01:36.763246 (+1845572us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0501 20:01:35.476590 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 20:01:35.516227 (+ 39637us) write_op.cc:424] Acquiring schema lock in shared mode
0501 20:01:35.516289 (+ 62us) write_op.cc:427] Acquired schema lock
0501 20:01:35.516302 (+ 13us) tablet.cc:660] Decoding operations
0501 20:01:35.539209 (+ 22907us) write_op.cc:612] Acquiring the partition lock for write op
0501 20:01:35.555202 (+ 15993us) write_op.cc:633] Partition lock acquired for write op
0501 20:01:35.558084 (+ 2882us) tablet.cc:683] Acquiring locks for 4 operations
0501 20:01:35.580557 (+ 22473us) tablet.cc:699] Row locks acquired
0501 20:01:35.580573 (+ 16us) write_op.cc:260] PREPARE: finished
0501 20:01:35.580741 (+ 168us) write_op.cc:270] Start()
0501 20:01:35.580875 (+ 134us) write_op.cc:276] Timestamp: P: 1714593695580682 usec, L: 0
0501 20:01:35.580906 (+ 31us) op_driver.cc:331] REPLICATION: starting
0501 20:01:35.581553 (+ 647us) log.cc:842] Serialized 1697 byte log entry
0501 20:01:36.268553 (+687000us) op_driver.cc:444] REPLICATION: finished
0501 20:01:36.455896 (+187343us) write_op.cc:301] APPLY: starting
0501 20:01:36.472394 (+ 16498us) tablet.cc:1365] starting BulkCheckPresence
0501 20:01:36.559721 (+ 87327us) tablet.cc:1368] finished BulkCheckPresence
0501 20:01:36.574704 (+ 14983us) tablet.cc:1370] starting ApplyRowOperation cycle
0501 20:01:36.617455 (+ 42751us) tablet.cc:1381] finished ApplyRowOperation cycle
0501 20:01:36.617716 (+ 261us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0501 20:01:36.617781 (+ 65us) write_op.cc:312] APPLY: finished
0501 20:01:36.618072 (+ 291us) log.cc:842] Serialized 40 byte log entry
0501 20:01:36.618252 (+ 180us) write_op.cc:481] Releasing partition, row and schema locks
0501 20:01:36.618342 (+ 90us) write_op.cc:446] Released schema lock
0501 20:01:36.618393 (+ 51us) write_op.cc:341] FINISH: Updating metrics
0501 20:01:36.618642 (+ 249us) write_op.cc:481] Releasing partition, row and schema locks
0501 20:01:36.618666 (+ 24us) write_op.cc:446] Released schema lock
Metrics: {"reported_tablets":6,"child_traces":[["op",{"apply.queue_time_us":187028,"apply.run_cpu_time_us":3376,"apply.run_wall_time_us":163052,"num_ops":4,"prepare.queue_time_us":213800,"prepare.run_cpu_time_us":32535,"prepare.run_wall_time_us":234554,"raft.queue_time_us":441983,"raft.run_cpu_time_us":40586,"raft.run_wall_time_us":159986,"replication_time_us":687524,"thread_start_us":660846,"threads_started":4,"wal-append.queue_time_us":221238}]]}
I20240501 20:01:36.802896 29801 raft_consensus.cc:1261] T 48630ae6eb394ac1b54955dd01cf0ed2 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:01:36.818212 29660 raft_consensus.cc:1226] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 2. Current term is 3. Ops: []
I20240501 20:01:36.818593 29652 raft_consensus.cc:1226] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 2. Current term is 3. Ops: []
I20240501 20:01:36.763005 31060 raft_consensus.cc:482] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:36.703614 30849 raft_consensus.cc:504] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:36.825652 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1a08b455b94ae089396a15c837ddc2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:36.828030 29806 raft_consensus.cc:3037] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:36.794390 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458740ab5c444b6ab3d1991c69db5084" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:36.839972 29729 raft_consensus.cc:1226] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 2. Current term is 3. Ops: []
I20240501 20:01:36.840243 29660 raft_consensus.cc:1226] T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 2. Current term is 3. Ops: []
I20240501 20:01:36.841379 29729 raft_consensus.cc:1226] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20240501 20:01:36.841671 29728 raft_consensus.cc:1226] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 3. Current term is 4. Ops: []
I20240501 20:01:36.836385 31043 consensus_queue.cc:1035] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20240501 20:01:36.850792 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5345f6eb62449ba69b4a9c2e349b31" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:36.817250 29651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcf30cf66714efdac997862dafbe4a2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:36.772135 30894 consensus_queue.cc:1035] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.069s
I20240501 20:01:36.863624 29734 raft_consensus.cc:2454] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 3.
I20240501 20:01:36.873750 29807 raft_consensus.cc:2454] T a8787bc4d6b44649b9138d906730110a P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 3.
I20240501 20:01:36.824201 30808 raft_consensus.cc:482] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:36.794586 30906 consensus_queue.cc:1046] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.044s
I20240501 20:01:36.883031 29625 leader_election.cc:302] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:36.889808 29656 raft_consensus.cc:1226] T 2d86f8848a184a87947fd927c44e5a35 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 2. Current term is 3. Ops: []
I20240501 20:01:36.898315 29653 raft_consensus.cc:2454] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 5.
I20240501 20:01:36.831355 31060 raft_consensus.cc:504] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:36.904481 29656 raft_consensus.cc:1226] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 2. Current term is 3. Ops: []
I20240501 20:01:36.889698 29805 raft_consensus.cc:2454] T 36057b3f24d44b9aac3a1378053a395f P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 3.
I20240501 20:01:36.886929 31052 consensus_queue.cc:1035] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20240501 20:01:36.914363 29656 raft_consensus.cc:1226] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:36.924885 31081 raft_consensus.cc:482] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:36.938915 29557 catalog_manager.cc:5484] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:36.892576 30808 raft_consensus.cc:504] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:36.918839 30646 raft_consensus.cc:2781] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:01:37.003472 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84fa28fb55f44b61ba1a88ed775a78ba" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:37.004284 29557 catalog_manager.cc:5484] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
W20240501 20:01:37.006896 29625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 123942us
I20240501 20:01:37.012475 29656 raft_consensus.cc:1226] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 2. Current term is 3. Ops: []
I20240501 20:01:37.006042 29557 catalog_manager.cc:5484] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 2 to 3, leader changed from ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 3 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:37.018379 29625 leader_election.cc:302] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
W20240501 20:01:37.014918 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 89218.6us
I20240501 20:01:37.049160 29557 catalog_manager.cc:5484] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 2 to 3, leader changed from ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 3 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
W20240501 20:01:37.013924 31075 log.cc:924] Time spent T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 0.097s user 0.000s sys 0.002s
W20240501 20:01:37.056080 30383 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 50429.9us
I20240501 20:01:37.056782 29557 catalog_manager.cc:5484] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:37.058373 29557 catalog_manager.cc:5484] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 1 to 2, leader changed from ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 2 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
W20240501 20:01:37.055747 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 244610us
I20240501 20:01:37.048929 30810 raft_consensus.cc:2781] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:01:37.060470 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d2cac25f354a97b1bfb53d57ae7c08" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:37.061427 29656 raft_consensus.cc:2379] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 3: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:37.062146 29698 leader_election.cc:302] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:37.065824 29557 catalog_manager.cc:5484] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 1 to 3, leader changed from ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 3 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:37.069293 29805 raft_consensus.cc:1261] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20240501 20:01:37.068289 29557 catalog_manager.cc:5484] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 1 to 2, leader changed from ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 2 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:37.068948 31017 raft_consensus.cc:504] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:37.069689 29805 pending_rounds.cc:76] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d: Aborting all ops after (but not including) 1
I20240501 20:01:37.069999 29805 pending_rounds.cc:97] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20240501 20:01:37.070291 29805 raft_consensus.cc:2866] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 20:01:37.070597 30849 leader_election.cc:288] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:37.070361 29557 catalog_manager.cc:5484] T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 1 to 2, leader changed from ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 2 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:37.071206 31017 leader_election.cc:288] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:37.071580 30886 raft_consensus.cc:2726] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 20:01:37.071869 29809 raft_consensus.cc:1226] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 2. Current term is 3. Ops: []
I20240501 20:01:37.066092 30808 leader_election.cc:288] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:37.022430 31081 raft_consensus.cc:504] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:37.076232 31081 leader_election.cc:288] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:37.076390 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5525f546426c47a38edf7f56ad04b7ab" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:37.093597 29659 raft_consensus.cc:2454] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 3.
I20240501 20:01:37.093888 29734 raft_consensus.cc:1226] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20240501 20:01:37.014757 30646 raft_consensus.cc:686] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:37.077489 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d88139f89c47ff84cad674147aae0f" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:37.111001 29653 raft_consensus.cc:2454] T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 2.
I20240501 20:01:37.047664 31045 raft_consensus.cc:482] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:37.114475 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec073fe645614fe28e7837de3c22510b" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:37.115290 29656 raft_consensus.cc:2362] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 3. Re-sending same reply.
W20240501 20:01:37.116166 29625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 60813.2us
I20240501 20:01:37.119048 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb27193abbf4644a48662909bc7b04a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:37.119474 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d2cac25f354a97b1bfb53d57ae7c08" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:37.122269 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07bdb8b09b8c4ef4b8e49daa9b8c01d6" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:37.123745 29734 raft_consensus.cc:1226] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 2. Current term is 3. Ops: []
I20240501 20:01:37.071913 29557 catalog_manager.cc:5484] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } }
W20240501 20:01:37.127178 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 77771us
I20240501 20:01:37.135310 29557 catalog_manager.cc:5484] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 1 to 2, leader changed from ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 2 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:37.078024 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8edcd903224d48f889b46e1b5318e0ec" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:37.082281 29806 raft_consensus.cc:2454] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 3.
I20240501 20:01:37.140148 29557 catalog_manager.cc:5484] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 1 to 3, leader changed from ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 3 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:37.140674 31081 raft_consensus.cc:482] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:37.144840 29800 raft_consensus.cc:1261] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240501 20:01:37.127766 30810 raft_consensus.cc:686] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:37.131978 30944 raft_consensus.cc:482] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:37.121882 30849 consensus_queue.cc:1046] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.028s
I20240501 20:01:37.145203 31060 leader_election.cc:288] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:37.154317 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8edcd903224d48f889b46e1b5318e0ec" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:37.162001 29803 raft_consensus.cc:2454] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 3.
I20240501 20:01:37.151175 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6797d82dda2948d8a7445fcd33e44752" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:37.151013 31017 raft_consensus.cc:482] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 888d99522e4844118f8f1cf26f268c8c)
I20240501 20:01:37.164710 29806 raft_consensus.cc:2396] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240501 20:01:37.166713 31017 raft_consensus.cc:504] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:37.170519 31017 leader_election.cc:288] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:37.121392 31045 raft_consensus.cc:504] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:37.141892 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6797d82dda2948d8a7445fcd33e44752" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:37.141844 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8787bc4d6b44649b9138d906730110a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:37.193218 29801 raft_consensus.cc:1226] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 2. Current term is 3. Ops: []
I20240501 20:01:37.131681 31019 consensus_queue.cc:1046] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.027s
I20240501 20:01:37.131347 30815 consensus_queue.cc:1046] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.026s
W20240501 20:01:37.202996 29625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 47328.8us
I20240501 20:01:37.160491 29728 raft_consensus.cc:1261] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:01:37.183426 30769 raft_consensus.cc:482] T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240501 20:01:37.209028 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 146924us
I20240501 20:01:37.148789 31081 raft_consensus.cc:504] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:37.215886 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 72103.4us
I20240501 20:01:37.216646 29624 leader_election.cc:302] T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:37.218161 29698 leader_election.cc:302] T c651044a305c4168805825a53d910899 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:37.206066 29658 raft_consensus.cc:2454] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 3.
I20240501 20:01:37.239511 29698 leader_election.cc:302] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:37.172112 30944 raft_consensus.cc:504] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:37.203359 30886 consensus_queue.cc:1035] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.016s
I20240501 20:01:37.217900 29727 raft_consensus.cc:2379] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 3: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:37.246194 29624 leader_election.cc:302] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:37.172336 30646 consensus_queue.cc:237] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:37.258343 29773 leader_election.cc:334] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 2.135s (SENT)
I20240501 20:01:37.260522 29773 leader_election.cc:302] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
W20240501 20:01:37.247278 31020 consensus_meta.cc:287] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d: Time spent flushing consensus metadata: real 0.610s user 0.010s sys 0.001s
I20240501 20:01:37.200114 31083 consensus_queue.cc:1035] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.063s
W20240501 20:01:37.250145 31042 consensus_meta.cc:287] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d: Time spent flushing consensus metadata: real 0.553s user 0.003s sys 0.007s
I20240501 20:01:37.274900 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6797d82dda2948d8a7445fcd33e44752" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
W20240501 20:01:37.250874 31044 consensus_meta.cc:287] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d: Time spent flushing consensus metadata: real 0.535s user 0.009s sys 0.002s
I20240501 20:01:37.219323 30753 consensus_queue.cc:1035] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20240501 20:01:37.268972 31020 raft_consensus.cc:504] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:37.264719 30815 raft_consensus.cc:2726] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:37.271055 31050 raft_consensus.cc:2726] T c651044a305c4168805825a53d910899 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 20:01:37.272104 31045 leader_election.cc:288] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:37.272856 30906 consensus_queue.cc:1035] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 20:01:37.274663 31042 raft_consensus.cc:504] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:37.214730 30769 raft_consensus.cc:504] T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:37.248945 31021 consensus_queue.cc:1046] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20240501 20:01:37.282291 31053 raft_consensus.cc:2781] T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:37.319272 30851 raft_consensus.cc:482] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:37.278360 29733 raft_consensus.cc:2396] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240501 20:01:37.282713 29730 raft_consensus.cc:2396] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20240501 20:01:37.361974 29732 raft_consensus.cc:1226] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 2. Current term is 3. Ops: []
W20240501 20:01:37.294723 31072 log.cc:924] Time spent T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 0.365s user 0.004s sys 0.010s
I20240501 20:01:37.242782 30810 consensus_queue.cc:237] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:37.330899 31044 raft_consensus.cc:2726] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20240501 20:01:37.292937 29625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 71625.5us
W20240501 20:01:37.362782 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 123377us
I20240501 20:01:37.342548 31083 raft_consensus.cc:2726] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:01:37.362257 31089 log.cc:924] Time spent T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 0.152s user 0.000s sys 0.002s
I20240501 20:01:37.356079 31053 raft_consensus.cc:482] T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:37.370620 29698 leader_election.cc:302] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:37.368140 30995 raft_consensus.cc:482] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:37.374049 30995 raft_consensus.cc:504] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
W20240501 20:01:37.375300 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 125349us
I20240501 20:01:37.376168 31044 raft_consensus.cc:2781] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:37.376731 31044 raft_consensus.cc:482] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:37.377192 31044 raft_consensus.cc:3037] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:37.377287 30414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e89c0572e4ee7be8e5315b459bdb3" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c" is_pre_election: true
W20240501 20:01:37.378996 31090 log.cc:924] Time spent T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 0.113s user 0.000s sys 0.000s
I20240501 20:01:37.377997 30414 raft_consensus.cc:2454] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 5.
I20240501 20:01:37.403756 31060 raft_consensus.cc:3032] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:01:37.391316 30849 raft_consensus.cc:3032] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:01:37.404429 31060 raft_consensus.cc:729] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:37.404518 30849 raft_consensus.cc:729] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:37.393426 30417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9046e38fc57f4364aa3d539e1fc01928" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c" is_pre_election: true
I20240501 20:01:37.395525 31042 leader_election.cc:288] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:37.352584 30650 consensus_queue.cc:1035] T 010ceabf3e5a4b1c9330b80dade765bb P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20240501 20:01:37.418185 30995 leader_election.cc:288] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:37.361901 30851 raft_consensus.cc:504] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:37.400473 30769 leader_election.cc:288] T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:37.402534 31081 leader_election.cc:288] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:37.403069 30944 leader_election.cc:288] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:37.390522 31017 raft_consensus.cc:3032] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:01:37.409955 29624 leader_election.cc:302] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:37.410766 31060 consensus_queue.cc:260] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:37.408447 30849 consensus_queue.cc:260] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:37.380816 31053 raft_consensus.cc:3037] T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:37.412179 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e89c0572e4ee7be8e5315b459bdb3" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:37.413242 31024 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:40656 (request call id 73) took 1922 ms. Trace:
I20240501 20:01:37.415038 29802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f5d1686214485d91a0baf4b870740e" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:37.415436 29651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9472c53f79ec423f82c396bd980674e2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:37.416097 30417 raft_consensus.cc:2454] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 2.
I20240501 20:01:37.417343 31020 leader_election.cc:288] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:37.419458 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9046e38fc57f4364aa3d539e1fc01928" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:37.427333 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 279743us
W20240501 20:01:37.433689 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 60368.9us
I20240501 20:01:37.444375 29735 raft_consensus.cc:1226] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 1. Current term is 3. Ops: []
I20240501 20:01:37.450670 29624 leader_election.cc:302] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:37.458330 29807 raft_consensus.cc:2454] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 888d99522e4844118f8f1cf26f268c8c in term 4.
I20240501 20:01:37.460647 29772 leader_election.cc:302] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 888d99522e4844118f8f1cf26f268c8c, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:37.444307 31042 raft_consensus.cc:482] T 010ceabf3e5a4b1c9330b80dade765bb P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:37.464524 29808 raft_consensus.cc:2454] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 888d99522e4844118f8f1cf26f268c8c in term 2.
I20240501 20:01:37.452096 31019 raft_consensus.cc:482] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:37.469446 29700 leader_election.cc:302] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:37.452925 30769 raft_consensus.cc:2726] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:01:37.449271 31017 raft_consensus.cc:729] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:37.414007 31044 raft_consensus.cc:504] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:37.477829 29624 leader_election.cc:302] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:37.481170 29700 leader_election.cc:302] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:37.481278 29698 leader_election.cc:302] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:37.473282 31020 raft_consensus.cc:2781] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240501 20:01:37.482533 31021 raft_consensus.cc:2726] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240501 20:01:37.480510 31083 raft_consensus.cc:482] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:37.485342 31043 consensus_queue.cc:1035] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20240501 20:01:37.482884 31087 raft_consensus.cc:2781] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:01:37.461025 31024 rpcz_store.cc:276] 0501 20:01:35.505828 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:35.508894 (+ 3066us) service_pool.cc:225] Handling call
0501 20:01:35.713300 (+204406us) ts_tablet_manager.cc:903] T 77be7101c0fe4085b6706296f609870a P 888d99522e4844118f8f1cf26f268c8c: Initiating tablet copy from peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
0501 20:01:36.731493 (+1018193us) tablet_copy_client.cc:353] Tablet copy session begun
0501 20:01:37.062776 (+331283us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:37.063053 (+ 277us) tablet_copy_client.cc:482] Wrote new tablet metadata
0501 20:01:37.412320 (+349267us) tablet_copy_client.cc:403] Wrote new consensus metadata
0501 20:01:37.412742 (+ 422us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:37.413099 (+ 357us) ts_tablet_manager.cc:958] Replica registered.
0501 20:01:37.413167 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":272,"tablet-copy.queue_time_us":140782,"thread_start_us":744961,"threads_started":5}
I20240501 20:01:37.495107 31024 tablet_copy_client.cc:783] T 77be7101c0fe4085b6706296f609870a P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Starting download of 0 data blocks...
I20240501 20:01:37.487165 31020 raft_consensus.cc:482] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 888d99522e4844118f8f1cf26f268c8c)
I20240501 20:01:37.496804 31020 raft_consensus.cc:3037] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:37.500247 31017 consensus_queue.cc:260] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:37.477042 31081 consensus_queue.cc:1035] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20240501 20:01:37.500898 31044 leader_election.cc:288] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:37.473676 31040 raft_consensus.cc:3032] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:01:37.480194 31098 consensus_queue.cc:1035] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20240501 20:01:37.476974 30849 raft_consensus.cc:3037] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:37.474539 31019 raft_consensus.cc:504] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:37.473573 31042 raft_consensus.cc:504] T 010ceabf3e5a4b1c9330b80dade765bb P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:37.509531 31019 leader_election.cc:288] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:37.510944 31042 leader_election.cc:288] T 010ceabf3e5a4b1c9330b80dade765bb P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:37.512499 31104 raft_consensus.cc:2781] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:01:37.512773 29698 leader_election.cc:302] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:37.513306 31104 raft_consensus.cc:686] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [term 3 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:37.514094 29698 leader_election.cc:302] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:37.514971 31045 raft_consensus.cc:2726] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240501 20:01:37.515250 29698 leader_election.cc:302] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:37.514660 31104 consensus_queue.cc:237] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } }
I20240501 20:01:37.472174 29651 raft_consensus.cc:2454] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 2.
I20240501 20:01:37.515664 31045 raft_consensus.cc:2781] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:01:37.487385 30851 leader_election.cc:288] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:37.525421 31045 raft_consensus.cc:686] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [term 3 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:37.491796 31083 raft_consensus.cc:504] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:37.526403 31045 consensus_queue.cc:237] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:37.527580 31083 leader_election.cc:288] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:37.516244 30894 raft_consensus.cc:2781] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:01:37.519480 31087 raft_consensus.cc:686] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [term 3 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:37.528637 30894 raft_consensus.cc:686] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [term 2 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:37.529196 31087 consensus_queue.cc:237] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:37.529505 30894 consensus_queue.cc:237] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:37.532230 31062 raft_consensus.cc:2726] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240501 20:01:37.491307 31107 raft_consensus.cc:482] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:37.533872 30839 raft_consensus.cc:3032] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:01:37.490196 30385 leader_election.cc:302] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 888d99522e4844118f8f1cf26f268c8c, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:37.534583 30839 raft_consensus.cc:729] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:37.534544 31107 raft_consensus.cc:504] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:37.536150 30385 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 60898us
I20240501 20:01:37.536890 31107 leader_election.cc:288] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:37.536937 30385 leader_election.cc:302] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 888d99522e4844118f8f1cf26f268c8c, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:37.492100 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e89c0572e4ee7be8e5315b459bdb3" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:37.539127 29660 raft_consensus.cc:2454] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 888d99522e4844118f8f1cf26f268c8c in term 5.
I20240501 20:01:37.485284 29772 leader_election.cc:302] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 888d99522e4844118f8f1cf26f268c8c, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:37.489519 31050 raft_consensus.cc:2781] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:37.517035 31040 raft_consensus.cc:729] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
W20240501 20:01:37.540477 29772 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.196:37227, user_credentials={real_user=slave}} blocked reactor thread for 55270us
W20240501 20:01:37.521538 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 43786us
I20240501 20:01:37.538280 31084 raft_consensus.cc:2781] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240501 20:01:37.551213 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e89c0572e4ee7be8e5315b459bdb3" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:37.495826 31024 tablet_copy_client.cc:647] T 77be7101c0fe4085b6706296f609870a P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Starting download of 1 WAL segments...
I20240501 20:01:37.539994 31050 raft_consensus.cc:482] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:37.561260 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6797d82dda2948d8a7445fcd33e44752" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:37.561450 31105 consensus_queue.cc:1035] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 20:01:37.561753 29624 leader_election.cc:302] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:37.561964 29806 raft_consensus.cc:2454] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 3.
I20240501 20:01:37.562953 29624 leader_election.cc:302] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:37.561472 30906 consensus_queue.cc:1035] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20240501 20:01:37.564203 31062 raft_consensus.cc:2726] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240501 20:01:37.564426 31063 raft_consensus.cc:2726] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240501 20:01:37.567569 29698 leader_election.cc:302] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:37.570123 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a35893b91247a99912eba787d787cf" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:37.491139 31021 raft_consensus.cc:482] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:37.580801 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49da4246f5f6438abe0e7e8c9a523d80" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:37.581699 29808 raft_consensus.cc:2454] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 4.
I20240501 20:01:37.585249 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9046e38fc57f4364aa3d539e1fc01928" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:37.551667 30815 raft_consensus.cc:2781] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:37.556591 31084 raft_consensus.cc:482] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:37.585722 29735 raft_consensus.cc:2454] T 9046e38fc57f4364aa3d539e1fc01928 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 888d99522e4844118f8f1cf26f268c8c in term 1.
I20240501 20:01:37.586405 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77be7101c0fe4085b6706296f609870a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:37.553050 31017 raft_consensus.cc:3037] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:37.588521 29698 leader_election.cc:302] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:37.548547 31108 raft_consensus.cc:2781] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:37.549157 31081 consensus_queue.cc:1046] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 2.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.042s
I20240501 20:01:37.563782 30935 consensus_queue.cc:1046] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20240501 20:01:37.595988 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfac893b4b4f457694a4eb184b6abb53" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:37.595774 29698 leader_election.cc:302] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:37.568253 31050 raft_consensus.cc:3037] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:37.535778 30839 consensus_queue.cc:260] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:37.601040 29808 raft_consensus.cc:2454] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 3.
I20240501 20:01:37.602367 29698 leader_election.cc:302] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:37.605989 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5525f546426c47a38edf7f56ad04b7ab" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:37.606158 31081 raft_consensus.cc:3032] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:01:37.606628 29808 raft_consensus.cc:3037] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:37.606767 31081 raft_consensus.cc:729] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:37.595140 30906 consensus_queue.cc:1035] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
I20240501 20:01:37.592586 31044 raft_consensus.cc:2781] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:01:37.609431 31044 raft_consensus.cc:686] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [term 3 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:37.566644 30886 consensus_queue.cc:1035] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20240501 20:01:37.590611 30815 raft_consensus.cc:482] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240501 20:01:37.611861 29625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 45739.3us
I20240501 20:01:37.593066 31084 raft_consensus.cc:3037] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:37.597002 31108 raft_consensus.cc:482] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:37.618706 29773 leader_election.cc:302] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 888d99522e4844118f8f1cf26f268c8c, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:37.619194 31101 raft_consensus.cc:2726] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:01:37.615345 31105 consensus_queue.cc:1159] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "fdfeac4190e045f290e003b2298d988d" responder_term: 3 status { last_received { term: 1 index: 1 } last_committed_idx: 1 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 2. Current term is 3. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20240501 20:01:37.620559 30810 raft_consensus.cc:3032] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:01:37.620754 31098 raft_consensus.cc:2726] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:37.622586 29624 leader_election.cc:302] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:37.638764 29625 leader_election.cc:302] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:37.586804 30753 consensus_queue.cc:1046] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
I20240501 20:01:37.624904 29700 leader_election.cc:302] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:37.639125 29773 leader_election.cc:302] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:37.587519 31021 raft_consensus.cc:504] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:37.610646 31044 consensus_queue.cc:237] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:37.612680 30894 raft_consensus.cc:2726] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 20:01:37.615664 30815 raft_consensus.cc:3037] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:37.602062 31045 consensus_queue.cc:1046] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20240501 20:01:37.608067 31081 consensus_queue.cc:260] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:37.633445 30839 raft_consensus.cc:3037] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:37.633201 31108 raft_consensus.cc:3037] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:37.660028 29700 leader_election.cc:302] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:37.660395 29651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1a08b455b94ae089396a15c837ddc2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:37.660929 31087 raft_consensus.cc:2781] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:37.593564 31040 consensus_queue.cc:260] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:37.661327 31087 raft_consensus.cc:482] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:37.661860 31087 raft_consensus.cc:3037] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:37.635551 30912 raft_consensus.cc:482] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:37.656208 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8787bc4d6b44649b9138d906730110a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:37.650786 31085 raft_consensus.cc:2726] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240501 20:01:37.649498 31053 raft_consensus.cc:504] T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:37.658213 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010ceabf3e5a4b1c9330b80dade765bb" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:37.637683 31019 raft_consensus.cc:2726] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:37.668534 29808 raft_consensus.cc:2454] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 4.
I20240501 20:01:37.635208 30810 raft_consensus.cc:729] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:37.649602 31060 raft_consensus.cc:3037] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:37.672196 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77be7101c0fe4085b6706296f609870a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:37.680698 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6797d82dda2948d8a7445fcd33e44752" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:37.681406 29653 raft_consensus.cc:2454] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 3.
I20240501 20:01:37.681640 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010ceabf3e5a4b1c9330b80dade765bb" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:37.673267 29624 leader_election.cc:302] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:37.671578 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfac893b4b4f457694a4eb184b6abb53" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:37.669304 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8edcd903224d48f889b46e1b5318e0ec" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:37.676198 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5525f546426c47a38edf7f56ad04b7ab" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:37.679162 29700 leader_election.cc:302] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:37.704265 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d88139f89c47ff84cad674147aae0f" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:37.704948 29808 raft_consensus.cc:3037] T f6d88139f89c47ff84cad674147aae0f P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 3
I20240501 20:01:37.679339 29806 raft_consensus.cc:1261] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20240501 20:01:37.654320 31101 raft_consensus.cc:2781] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:37.702844 29659 raft_consensus.cc:2454] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 3.
I20240501 20:01:37.702390 29660 raft_consensus.cc:3037] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:37.669198 31107 raft_consensus.cc:2781] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240501 20:01:37.678361 31053 leader_election.cc:288] T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:37.718570 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 45379.5us
I20240501 20:01:37.712791 31101 raft_consensus.cc:482] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:37.719058 31099 raft_consensus.cc:2726] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:37.719358 30753 raft_consensus.cc:2781] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader election won for term 4
W20240501 20:01:37.665477 31106 consensus_queue.cc:1159] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ee0d97247d394a8facfa774f15f3a67d" responder_term: 3 status { last_received { term: 3 index: 2 } last_committed_idx: 2 last_received_current_leader { term: 3 index: 2 } }
I20240501 20:01:37.706178 31019 raft_consensus.cc:2741] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20240501 20:01:37.674512 31083 consensus_queue.cc:1035] T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20240501 20:01:37.716903 31107 raft_consensus.cc:482] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:37.672446 30912 raft_consensus.cc:504] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:37.721860 31101 raft_consensus.cc:3037] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Advancing to term 4
W20240501 20:01:37.766866 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 44938.7us
I20240501 20:01:37.708235 31020 raft_consensus.cc:504] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:37.761490 31107 raft_consensus.cc:3037] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:37.781368 29808 raft_consensus.cc:2454] T f6d88139f89c47ff84cad674147aae0f P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 3.
I20240501 20:01:37.738277 31045 consensus_queue.cc:1035] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
I20240501 20:01:37.795946 29772 leader_election.cc:302] T f6d88139f89c47ff84cad674147aae0f P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:37.796996 31019 raft_consensus.cc:2741] T f6d88139f89c47ff84cad674147aae0f P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20240501 20:01:37.797125 29772 leader_election.cc:302] T ec073fe645614fe28e7837de3c22510b P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:37.798804 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f5d1686214485d91a0baf4b870740e" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:37.799960 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49da4246f5f6438abe0e7e8c9a523d80" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:37.800122 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a35893b91247a99912eba787d787cf" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:37.804267 31019 raft_consensus.cc:2781] T ec073fe645614fe28e7837de3c22510b P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:01:37.804668 29772 leader_election.cc:302] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:37.804960 31019 raft_consensus.cc:686] T ec073fe645614fe28e7837de3c22510b P ee0d97247d394a8facfa774f15f3a67d [term 3 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:37.795778 29660 raft_consensus.cc:2454] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 4.
I20240501 20:01:37.762795 30753 raft_consensus.cc:686] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:37.810045 29809 raft_consensus.cc:2396] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240501 20:01:37.769601 29658 raft_consensus.cc:1261] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240501 20:01:37.866694 29625 leader_election.cc:302] T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:37.888572 31118 raft_consensus.cc:2781] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:37.889221 31118 raft_consensus.cc:482] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:37.889664 31118 raft_consensus.cc:3037] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:37.774256 30417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e89c0572e4ee7be8e5315b459bdb3" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c" is_pre_election: true
I20240501 20:01:37.792038 31043 raft_consensus.cc:3032] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [term 2 LEADER]: Stepping down as leader of term 2
W20240501 20:01:37.904338 29772 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 98644.2us
I20240501 20:01:37.905062 29772 leader_election.cc:302] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:37.749073 30810 consensus_queue.cc:260] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:37.773855 31063 consensus_queue.cc:1046] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.127s
I20240501 20:01:37.878471 30769 raft_consensus.cc:482] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240501 20:01:37.913997 29625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 47426us
I20240501 20:01:37.806128 31019 consensus_queue.cc:237] T ec073fe645614fe28e7837de3c22510b P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:37.904575 31043 raft_consensus.cc:729] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:37.937294 29807 raft_consensus.cc:1226] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 2. Current term is 3. Ops: []
I20240501 20:01:37.942130 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:37.937844 30650 raft_consensus.cc:2781] T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:01:37.907330 31021 leader_election.cc:288] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:38.010821 29625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 45220.5us
W20240501 20:01:37.962424 31125 log.cc:924] Time spent T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.212s user 0.001s sys 0.000s
W20240501 20:01:38.007673 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 56472.6us
I20240501 20:01:37.960270 30810 raft_consensus.cc:3037] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:37.923420 30769 raft_consensus.cc:504] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:38.024462 31020 leader_election.cc:288] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:37.964004 30650 raft_consensus.cc:686] T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
W20240501 20:01:38.036908 30849 consensus_meta.cc:287] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d: Time spent flushing consensus metadata: real 0.530s user 0.006s sys 0.005s
W20240501 20:01:38.038652 31119 log.cc:924] Time spent T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 0.321s user 0.000s sys 0.001s
W20240501 20:01:38.057909 31127 log.cc:924] Time spent T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d: Append to log took a long time: real 0.162s user 0.001s sys 0.000s
I20240501 20:01:37.948642 30753 consensus_queue.cc:237] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:38.064728 30650 consensus_queue.cc:237] T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:38.065555 30414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e89c0572e4ee7be8e5315b459bdb3" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c"
I20240501 20:01:38.011541 31106 consensus_queue.cc:1046] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
I20240501 20:01:38.066146 31101 raft_consensus.cc:504] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:38.067087 31132 ts_tablet_manager.cc:902] T be383be783234855878cb8fd442c7b05 P 888d99522e4844118f8f1cf26f268c8c: Initiating tablet copy from peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
W20240501 20:01:38.057250 31126 log.cc:924] Time spent T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.247s user 0.000s sys 0.001s
I20240501 20:01:38.046425 31020 raft_consensus.cc:2781] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:38.093688 31108 raft_consensus.cc:504] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:38.080279 29624 leader_election.cc:302] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
W20240501 20:01:38.160710 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 80517.9us
W20240501 20:01:38.173388 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 61272.9us
W20240501 20:01:38.157681 29625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 44092.3us
I20240501 20:01:38.053268 30912 leader_election.cc:288] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:38.082187 31098 raft_consensus.cc:3032] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:01:38.111383 31020 raft_consensus.cc:482] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:38.109797 31085 raft_consensus.cc:482] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:38.227732 29809 raft_consensus.cc:1261] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240501 20:01:38.232312 30808 raft_consensus.cc:482] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:38.244720 31129 consensus_queue.cc:1035] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20240501 20:01:38.245162 31138 log.cc:924] Time spent T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 0.180s user 0.000s sys 0.001s
I20240501 20:01:38.246372 31108 leader_election.cc:288] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 3 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:38.193169 31098 raft_consensus.cc:729] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
W20240501 20:01:38.247910 30815 consensus_meta.cc:287] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d: Time spent flushing consensus metadata: real 0.593s user 0.003s sys 0.004s
I20240501 20:01:38.227104 31118 raft_consensus.cc:504] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:38.082820 31084 raft_consensus.cc:504] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:38.268388 31118 leader_election.cc:288] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:38.233368 30769 leader_election.cc:288] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:38.239215 31020 raft_consensus.cc:3037] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:38.269327 31081 raft_consensus.cc:3037] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:38.271545 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb27193abbf4644a48662909bc7b04a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:38.276165 29653 raft_consensus.cc:2379] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 3: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:38.277309 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e89c0572e4ee7be8e5315b459bdb3" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:38.280768 29653 raft_consensus.cc:2379] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 5: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
W20240501 20:01:38.262972 31050 consensus_meta.cc:287] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d: Time spent flushing consensus metadata: real 0.649s user 0.009s sys 0.001s
I20240501 20:01:38.282335 29772 leader_election.cc:302] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, 888d99522e4844118f8f1cf26f268c8c
W20240501 20:01:38.259238 30839 consensus_meta.cc:287] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f: Time spent flushing consensus metadata: real 0.605s user 0.009s sys 0.001s
W20240501 20:01:38.281476 29655 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.561s user 0.015s sys 0.011s
W20240501 20:01:38.283537 29655 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.563s user 0.016s sys 0.011s
I20240501 20:01:38.284447 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e89c0572e4ee7be8e5315b459bdb3" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:38.290001 29653 raft_consensus.cc:2379] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 5: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:38.262913 30912 raft_consensus.cc:482] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:38.296514 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac2171683884f6f97cdfeb0a656cb49" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:38.297470 29653 raft_consensus.cc:2454] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 3.
I20240501 20:01:38.298722 29772 leader_election.cc:302] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
W20240501 20:01:38.289791 29773 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.643s user 0.007s sys 0.019s
W20240501 20:01:38.302155 29773 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.656s user 0.007s sys 0.019s
W20240501 20:01:38.302755 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 663702us
I20240501 20:01:38.304431 29773 leader_election.cc:302] T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:38.257120 30808 raft_consensus.cc:504] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:38.283901 31118 raft_consensus.cc:482] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:38.330569 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be383be783234855878cb8fd442c7b05" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:38.331586 31079 raft_consensus.cc:482] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:38.334913 31103 raft_consensus.cc:482] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:38.334664 31079 raft_consensus.cc:504] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:38.273907 30815 raft_consensus.cc:504] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:38.248342 31085 raft_consensus.cc:504] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:38.335407 31103 raft_consensus.cc:504] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:38.346572 31107 raft_consensus.cc:504] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:38.347487 31103 leader_election.cc:288] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:38.307499 31060 raft_consensus.cc:2781] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:38.348205 31060 raft_consensus.cc:482] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:38.348587 31060 raft_consensus.cc:3037] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:38.343890 31091 raft_consensus.cc:482] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:38.336076 30417 raft_consensus.cc:2379] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 6: Already voted for candidate 888d99522e4844118f8f1cf26f268c8c in this term.
I20240501 20:01:38.354565 29773 leader_election.cc:302] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 888d99522e4844118f8f1cf26f268c8c, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:38.291021 31050 raft_consensus.cc:504] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:38.357023 29773 leader_election.cc:302] T d1d2cac25f354a97b1bfb53d57ae7c08 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:38.352815 29653 raft_consensus.cc:3037] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:38.305076 30912 raft_consensus.cc:504] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } }
I20240501 20:01:38.357017 31091 raft_consensus.cc:504] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:38.344363 31135 raft_consensus.cc:2726] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240501 20:01:38.349025 30808 leader_election.cc:288] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:38.353003 29651 raft_consensus.cc:2396] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240501 20:01:38.342306 31105 raft_consensus.cc:2726] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:38.362512 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6797d82dda2948d8a7445fcd33e44752" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:38.365314 29803 raft_consensus.cc:2379] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d in current term 4: Already voted for candidate ee0d97247d394a8facfa774f15f3a67d in this term.
I20240501 20:01:38.337450 31118 raft_consensus.cc:504] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:38.357355 31101 leader_election.cc:288] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:38.360523 30944 raft_consensus.cc:482] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:38.340526 31098 consensus_queue.cc:260] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:38.381624 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8cd40cb4044d18a86001ce6ddf97ab" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:38.380951 31130 log.cc:924] Time spent T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 0.401s user 0.010s sys 0.003s
I20240501 20:01:38.473306 31085 leader_election.cc:288] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:38.474376 29651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49da4246f5f6438abe0e7e8c9a523d80" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:38.395890 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48fd0a8e289401aa7417c42ff2c81b2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:38.475409 29651 raft_consensus.cc:2379] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 5: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:38.379153 31118 leader_election.cc:288] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:38.476011 31118 raft_consensus.cc:2726] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 20:01:38.262166 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
W20240501 20:01:38.476951 31017 consensus_meta.cc:287] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d: Time spent flushing consensus metadata: real 0.879s user 0.011s sys 0.000s
I20240501 20:01:38.477692 29805 raft_consensus.cc:1261] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 20:01:38.477975 30912 leader_election.cc:288] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:38.477707 31017 raft_consensus.cc:3037] T d1d2cac25f354a97b1bfb53d57ae7c08 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Advancing to term 3
W20240501 20:01:38.478873 31137 log.cc:924] Time spent T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 0.405s user 0.000s sys 0.012s
I20240501 20:01:38.478572 29558 catalog_manager.cc:5484] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c reported cstate change: term changed from 1 to 3, leader changed from ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 3 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:38.480346 29623 leader_election.cc:302] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: 888d99522e4844118f8f1cf26f268c8c, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:38.419472 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8edcd903224d48f889b46e1b5318e0ec" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:38.452973 30417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9046e38fc57f4364aa3d539e1fc01928" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c"
I20240501 20:01:38.380520 29773 leader_election.cc:302] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:38.371563 31091 leader_election.cc:288] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:38.369675 31135 raft_consensus.cc:2726] T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240501 20:01:38.452798 31087 consensus_meta.cc:287] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d: Time spent flushing consensus metadata: real 0.790s user 0.009s sys 0.003s
I20240501 20:01:38.404927 31040 raft_consensus.cc:3037] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Advancing to term 4
W20240501 20:01:38.512001 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 131549us
I20240501 20:01:38.394379 31079 leader_election.cc:288] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:38.418116 31050 leader_election.cc:288] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:38.492944 30417 raft_consensus.cc:2379] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 3: Already voted for candidate 888d99522e4844118f8f1cf26f268c8c in this term.
I20240501 20:01:38.416327 31084 leader_election.cc:288] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 6 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:38.540921 31020 raft_consensus.cc:504] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:38.544162 31098 raft_consensus.cc:3037] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:38.491930 31085 raft_consensus.cc:3037] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Advancing to term 6
W20240501 20:01:38.545936 29657 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.534s user 0.004s sys 0.035s
I20240501 20:01:38.546967 29773 leader_election.cc:302] T 3c5345f6eb62449ba69b4a9c2e349b31 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:38.416325 29654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c651044a305c4168805825a53d910899" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:38.548508 29653 raft_consensus.cc:2454] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 3.
W20240501 20:01:38.565459 29657 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.554s user 0.005s sys 0.035s
W20240501 20:01:38.544982 29652 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.645s user 0.000s sys 0.031s
W20240501 20:01:38.517572 31131 log.cc:924] Time spent T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.501s user 0.000s sys 0.012s
W20240501 20:01:38.567600 29652 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.668s user 0.000s sys 0.032s
I20240501 20:01:38.524006 29651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e89c0572e4ee7be8e5315b459bdb3" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:38.546108 31107 leader_election.cc:288] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:38.406297 31043 consensus_queue.cc:260] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:38.481523 30944 raft_consensus.cc:504] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
W20240501 20:01:38.735685 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 170825us
W20240501 20:01:38.569461 29658 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.548s user 0.008s sys 0.037s
I20240501 20:01:38.572132 31135 raft_consensus.cc:2726] T 3c5345f6eb62449ba69b4a9c2e349b31 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:01:38.516150 31087 raft_consensus.cc:504] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:38.740414 29626 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.662s user 0.008s sys 0.018s
W20240501 20:01:38.817041 29626 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.920s user 0.009s sys 0.023s
I20240501 20:01:38.557842 30912 raft_consensus.cc:482] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:38.775669 29654 raft_consensus.cc:2454] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 2.
I20240501 20:01:38.733314 31079 raft_consensus.cc:482] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
W20240501 20:01:38.793056 29658 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.771s user 0.008s sys 0.037s
I20240501 20:01:38.544765 31118 raft_consensus.cc:2726] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:38.830035 31052 raft_consensus.cc:482] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:38.831653 29772 leader_election.cc:302] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:38.785730 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69915dd5992641e3a24f7f45f1cd5c0c" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:38.792665 31021 raft_consensus.cc:482] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
W20240501 20:01:38.843544 29801 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.178s user 0.026s sys 0.009s
W20240501 20:01:38.846874 29801 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.182s user 0.027s sys 0.009s
I20240501 20:01:38.819147 31106 raft_consensus.cc:482] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:38.821125 31043 raft_consensus.cc:3037] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:38.798576 30944 leader_election.cc:288] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:38.844821 29657 raft_consensus.cc:2454] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 3.
I20240501 20:01:38.861126 29772 leader_election.cc:302] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:38.868827 29657 raft_consensus.cc:1226] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20240501 20:01:38.869758 29657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59142 (request call id 885) took 1167 ms. Trace:
I20240501 20:01:38.870275 29657 rpcz_store.cc:276] 0501 20:01:37.702637 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:38.865515 (+1162878us) service_pool.cc:225] Handling call
0501 20:01:38.868516 (+ 3001us) raft_consensus.cc:1457] Updating replica for 1 ops
0501 20:01:38.869436 (+ 920us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:38.869692 (+ 256us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:38.870436 31143 consensus_queue.cc:1046] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 20:01:38.830993 29654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6797d82dda2948d8a7445fcd33e44752" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:38.797116 31101 consensus_queue.cc:1035] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.063s
I20240501 20:01:38.844020 31107 raft_consensus.cc:482] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 888d99522e4844118f8f1cf26f268c8c)
I20240501 20:01:38.873423 29801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41858 (request call id 902) took 1192 ms. Trace:
I20240501 20:01:38.871546 29655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59142 (request call id 870) took 1204 ms. Trace:
I20240501 20:01:38.842682 31105 raft_consensus.cc:482] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:38.842909 31083 raft_consensus.cc:482] T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:38.839402 31060 raft_consensus.cc:504] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:38.880825 29701 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.955s user 0.020s sys 0.035s
W20240501 20:01:38.894256 29701 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.018s user 0.020s sys 0.035s
I20240501 20:01:38.843971 31118 raft_consensus.cc:482] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240501 20:01:38.848860 31137 log.cc:924] Time spent T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 0.057s user 0.000s sys 0.001s
I20240501 20:01:38.839928 30912 raft_consensus.cc:504] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:38.892112 29655 rpcz_store.cc:276] 0501 20:01:37.667100 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:37.667272 (+ 172us) service_pool.cc:225] Handling call
0501 20:01:37.667686 (+ 414us) raft_consensus.cc:1457] Updating replica for 1 ops
0501 20:01:37.719562 (+ 51876us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:37.719588 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0501 20:01:37.720060 (+ 472us) log.cc:842] Serialized 26 byte log entry
0501 20:01:38.283981 (+563921us) raft_consensus.cc:1631] Marking committed up to 2
0501 20:01:38.337643 (+ 53662us) log.cc:842] Serialized 14 byte log entry
0501 20:01:38.341826 (+ 4183us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:38.341907 (+ 81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0501 20:01:38.871343 (+529436us) raft_consensus.cc:1671] finished
0501 20:01:38.871365 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:38.871492 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":75852,"prepare.queue_time_us":52300,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":2233,"raft.queue_time_us":51005,"raft.run_cpu_time_us":284,"raft.run_wall_time_us":23961,"thread_start_us":563549,"threads_started":1,"wal-append.queue_time_us":837242}
I20240501 20:01:38.893069 31105 raft_consensus.cc:504] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:38.904472 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfac893b4b4f457694a4eb184b6abb53" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:38.905125 29805 raft_consensus.cc:3037] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:38.886835 29654 raft_consensus.cc:3037] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:38.908607 29655 raft_consensus.cc:1226] T 2d86f8848a184a87947fd927c44e5a35 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 2. Current term is 3. Ops: []
I20240501 20:01:38.859306 31021 raft_consensus.cc:504] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:38.859514 31052 raft_consensus.cc:504] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:38.893144 31083 raft_consensus.cc:504] T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:38.892714 31087 leader_election.cc:288] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:38.865514 31079 raft_consensus.cc:504] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:38.889834 31107 raft_consensus.cc:504] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:38.933315 29735 raft_consensus.cc:2396] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240501 20:01:38.914979 31020 leader_election.cc:288] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:38.872267 31106 raft_consensus.cc:504] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:38.901278 31118 raft_consensus.cc:504] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:38.932940 31081 consensus_meta.cc:287] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d: Time spent flushing consensus metadata: real 0.658s user 0.006s sys 0.006s
W20240501 20:01:38.944988 29698 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.068s user 0.014s sys 0.058s
I20240501 20:01:38.950251 29652 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59142 (request call id 880) took 1274 ms. Trace:
I20240501 20:01:38.950682 29652 rpcz_store.cc:276] 0501 20:01:37.675717 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:37.707320 (+ 31603us) service_pool.cc:225] Handling call
0501 20:01:37.707637 (+ 317us) raft_consensus.cc:1457] Updating replica for 1 ops
0501 20:01:37.772523 (+ 64886us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:37.772552 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0501 20:01:37.772977 (+ 425us) log.cc:842] Serialized 26 byte log entry
0501 20:01:38.568156 (+795179us) raft_consensus.cc:1631] Marking committed up to 2
0501 20:01:38.736743 (+168587us) log.cc:842] Serialized 14 byte log entry
0501 20:01:38.790058 (+ 53315us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:38.790453 (+ 395us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0501 20:01:38.950039 (+159586us) raft_consensus.cc:1671] finished
0501 20:01:38.950062 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:38.950193 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":169579,"prepare.queue_time_us":172321,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":120788,"raft.run_cpu_time_us":302,"raft.run_wall_time_us":36602,"thread_start_us":668237,"threads_started":1,"wal-append.queue_time_us":1141447}
I20240501 20:01:38.953238 31106 leader_election.cc:288] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:38.953440 31118 leader_election.cc:288] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:38.953931 31105 leader_election.cc:288] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:38.954149 31118 raft_consensus.cc:482] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:38.955087 31042 raft_consensus.cc:482] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:38.954778 31118 raft_consensus.cc:504] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:38.955592 31042 raft_consensus.cc:504] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:38.957720 31118 leader_election.cc:288] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:38.958235 31118 raft_consensus.cc:2781] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:38.958624 31118 raft_consensus.cc:482] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:38.958715 31085 raft_consensus.cc:2726] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20240501 20:01:38.960845 29698 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.084s user 0.014s sys 0.058s
I20240501 20:01:38.967083 29805 raft_consensus.cc:2454] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 4.
W20240501 20:01:38.968989 29808 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.899s user 0.002s sys 0.058s
I20240501 20:01:38.891979 29801 rpcz_store.cc:276] 0501 20:01:37.655681 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:37.655869 (+ 188us) service_pool.cc:225] Handling call
0501 20:01:37.664575 (+ 8706us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:37.664766 (+ 191us) raft_consensus.cc:1520] Early marking committed up to index 2
0501 20:01:37.665055 (+ 289us) log.cc:842] Serialized 14 byte log entry
0501 20:01:38.847362 (+1182307us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:38.847414 (+ 52us) raft_consensus.cc:1631] Marking committed up to 2
0501 20:01:38.847453 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:38.847514 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:38.847628 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":509,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":87481,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":8991,"thread_start_us":1182054,"threads_started":1}
W20240501 20:01:38.969331 29808 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.900s user 0.002s sys 0.058s
I20240501 20:01:38.974965 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:38.979534 31107 leader_election.cc:288] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:38.942566 31020 raft_consensus.cc:2781] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:38.982560 29700 leader_election.cc:302] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:38.959158 31118 raft_consensus.cc:3037] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Advancing to term 4
W20240501 20:01:38.989032 29660 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.915s user 0.014s sys 0.025s
W20240501 20:01:38.991793 29660 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.918s user 0.014s sys 0.026s
I20240501 20:01:38.955103 30912 leader_election.cc:288] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:38.958715 31017 raft_consensus.cc:2726] T d1d2cac25f354a97b1bfb53d57ae7c08 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:01:38.963174 31063 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.027s user 0.006s sys 0.031s
W20240501 20:01:38.988857 31019 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.928s user 0.032s sys 0.032s
W20240501 20:01:38.992436 29659 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.078s user 0.027s sys 0.041s
I20240501 20:01:38.987084 31020 raft_consensus.cc:482] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240501 20:01:39.001720 29656 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.053s user 0.014s sys 0.045s
I20240501 20:01:38.980504 31087 raft_consensus.cc:482] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:38.990757 31081 raft_consensus.cc:2781] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:01:38.993031 31106 raft_consensus.cc:482] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
W20240501 20:01:39.009941 29659 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.095s user 0.027s sys 0.041s
I20240501 20:01:39.022477 29659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59142 (request call id 881) took 1346 ms. Trace:
W20240501 20:01:39.015825 29625 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.809s user 0.056s sys 0.112s
I20240501 20:01:39.011297 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84fa28fb55f44b61ba1a88ed775a78ba" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:38.996371 31043 raft_consensus.cc:482] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:38.995306 31101 raft_consensus.cc:482] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:39.020407 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05df43bdfa479b902b1da297fb5974" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
W20240501 20:01:39.024493 29625 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.818s user 0.056s sys 0.112s
I20240501 20:01:39.024739 29731 raft_consensus.cc:2396] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
W20240501 20:01:39.027606 29699 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.060s user 0.020s sys 0.065s
W20240501 20:01:39.014245 29656 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.069s user 0.014s sys 0.045s
W20240501 20:01:39.027963 29699 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.151s user 0.021s sys 0.065s
W20240501 20:01:39.002004 30849 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.769s user 0.010s sys 0.031s
I20240501 20:01:39.028615 29733 raft_consensus.cc:1226] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [term 3 LEADER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 2. Current term is 3. Ops: []
I20240501 20:01:39.029493 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6797d82dda2948d8a7445fcd33e44752" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:39.000777 31021 leader_election.cc:288] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:39.035430 29730 raft_consensus.cc:2379] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 4: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:39.016086 31020 raft_consensus.cc:3037] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:39.028383 29659 rpcz_store.cc:276] 0501 20:01:37.675969 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:37.710830 (+ 34861us) service_pool.cc:225] Handling call
0501 20:01:37.711208 (+ 378us) raft_consensus.cc:1457] Updating replica for 1 ops
0501 20:01:37.749099 (+ 37891us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:37.749153 (+ 54us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0501 20:01:37.749657 (+ 504us) log.cc:842] Serialized 26 byte log entry
0501 20:01:37.873056 (+123399us) raft_consensus.cc:1631] Marking committed up to 2
0501 20:01:37.900702 (+ 27646us) log.cc:842] Serialized 14 byte log entry
0501 20:01:37.937461 (+ 36759us) spinlock_profiling.cc:243] Waited 2.12 ms on lock 0x7b4800080058. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe45c723431 00007fe45c71a527 00007fe45c71a3ba 00007fe45c80e57f 00007fe45c80e247 00007fe45c82274c 00007fe45c8226f8 00007fe45c822678 00007fe45c822633 00007fe45c8218dc 00007fe45b7a49b6
0501 20:01:38.231283 (+293822us) spinlock_profiling.cc:243] Waited 10.4 ms on lock 0x7b240017ce28. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe454797b51 00007fe4547ac024 00007fe4547aa13b 00007fe454838149 00007fe454837dd5 00007fe454837330 00007fe45752a3a8 00007fe45752ab08 00007fe457502618 00007fe4574fda1a 00007fe4574fce66
0501 20:01:39.022160 (+790877us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:39.022244 (+ 84us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0501 20:01:39.022286 (+ 42us) raft_consensus.cc:1671] finished
0501 20:01:39.022301 (+ 15us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:39.022426 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":90525,"prepare.queue_time_us":48624,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":80676,"raft.run_cpu_time_us":6323,"raft.run_wall_time_us":176859,"spinlock_wait_cycles":28782720,"thread_start_us":1205664,"threads_started":2,"wal-append.queue_time_us":198669}
I20240501 20:01:39.014775 31060 leader_election.cc:288] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:39.015384 31017 raft_consensus.cc:482] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 888d99522e4844118f8f1cf26f268c8c)
I20240501 20:01:39.001448 30912 raft_consensus.cc:482] T 36057b3f24d44b9aac3a1378053a395f P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240501 20:01:39.006095 31019 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.946s user 0.032s sys 0.033s
W20240501 20:01:39.016139 31132 thread.cc:636] tablet-download-pool-be383be783234855878cb8fd442c7b05 [worker] (thread pool) Time spent creating pthread: real 0.895s user 0.026s sys 0.037s
W20240501 20:01:39.001430 31063 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.071s user 0.006s sys 0.031s
I20240501 20:01:39.035066 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be383be783234855878cb8fd442c7b05" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:39.043740 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb27193abbf4644a48662909bc7b04a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
W20240501 20:01:39.036358 30849 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.803s user 0.010s sys 0.032s
I20240501 20:01:39.030755 30417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ed27033ab1645e9ab0600ab635d9589" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c" is_pre_election: true
I20240501 20:01:39.046523 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49da4246f5f6438abe0e7e8c9a523d80" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:39.037958 31147 raft_consensus.cc:482] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:39.063513 31150 raft_consensus.cc:482] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:39.063195 31147 raft_consensus.cc:504] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:39.063999 31150 raft_consensus.cc:504] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:39.055174 29801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f261d2a05bf4c838532049601694b4a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:39.045375 31152 raft_consensus.cc:482] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:39.022308 31087 raft_consensus.cc:504] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:39.043613 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31fff64b453a423eb0f567b3171564fb" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:39.043850 31091 raft_consensus.cc:482] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:39.033479 31043 raft_consensus.cc:504] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:39.033965 31101 raft_consensus.cc:504] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:39.057090 29558 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.549s user 0.011s sys 0.009s
I20240501 20:01:39.035965 31106 raft_consensus.cc:504] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:39.045687 31083 leader_election.cc:288] T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:39.057138 31132 thread.cc:602] tablet-download-pool-be383be783234855878cb8fd442c7b05 [worker] (thread pool) Time spent starting thread: real 0.948s user 0.026s sys 0.038s
W20240501 20:01:39.030326 29732 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.787s user 0.015s sys 0.018s
I20240501 20:01:39.051448 31079 leader_election.cc:288] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:39.081156 29734 raft_consensus.cc:2454] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
I20240501 20:01:39.074299 31152 raft_consensus.cc:504] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:39.082989 31106 leader_election.cc:288] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:39.083428 31152 leader_election.cc:288] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:39.087595 31087 leader_election.cc:288] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:39.093637 29808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41864 (request call id 958) took 1144 ms. Trace:
W20240501 20:01:39.062458 29802 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.696s user 0.017s sys 0.003s
W20240501 20:01:39.079232 29732 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.836s user 0.015s sys 0.018s
I20240501 20:01:39.079180 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d88139f89c47ff84cad674147aae0f" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:39.069183 31147 leader_election.cc:288] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:39.072427 30849 raft_consensus.cc:3032] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 3 LEADER]: Stepping down as leader of term 3
W20240501 20:01:39.076390 29558 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.568s user 0.012s sys 0.009s
W20240501 20:01:39.069995 30650 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.697s user 0.003s sys 0.012s
I20240501 20:01:39.098210 29801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a2bba62f434d1c8489b55c8fec2187" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:39.102946 30935 raft_consensus.cc:3032] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:01:39.076300 31091 raft_consensus.cc:504] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:39.055071 30912 raft_consensus.cc:504] T 36057b3f24d44b9aac3a1378053a395f P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:39.062372 31017 raft_consensus.cc:504] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:39.094134 29808 rpcz_store.cc:276] 0501 20:01:37.948883 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:37.949511 (+ 628us) service_pool.cc:225] Handling call
0501 20:01:37.965634 (+ 16123us) raft_consensus.cc:1457] Updating replica for 1 ops
0501 20:01:38.015838 (+ 50204us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:38.015880 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0501 20:01:38.016432 (+ 552us) log.cc:842] Serialized 26 byte log entry
0501 20:01:38.969733 (+953301us) raft_consensus.cc:1631] Marking committed up to 1
0501 20:01:38.969775 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:38.969839 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0501 20:01:39.093378 (+123539us) raft_consensus.cc:1671] finished
0501 20:01:39.093400 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:39.093559 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5372,"raft.queue_time_us":62723,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":8874,"thread_start_us":900174,"threads_started":1,"wal-append.queue_time_us":1044155}
W20240501 20:01:39.078488 30753 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.681s user 0.010s sys 0.005s
I20240501 20:01:39.110749 29804 raft_consensus.cc:2379] T f6d88139f89c47ff84cad674147aae0f P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d in current term 3: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:39.073647 31151 raft_consensus.cc:482] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240501 20:01:39.113359 30650 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.742s user 0.003s sys 0.012s
I20240501 20:01:39.118065 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05df43bdfa479b902b1da297fb5974" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:39.092238 29728 raft_consensus.cc:2454] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 3.
I20240501 20:01:39.118901 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84fa28fb55f44b61ba1a88ed775a78ba" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:39.107131 29802 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.742s user 0.017s sys 0.003s
W20240501 20:01:39.117470 30906 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.636s user 0.008s sys 0.024s
I20240501 20:01:39.130124 29803 raft_consensus.cc:2454] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 4.
I20240501 20:01:39.112003 30849 raft_consensus.cc:729] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:39.115334 30935 raft_consensus.cc:729] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:39.134395 29700 leader_election.cc:302] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
W20240501 20:01:39.136252 29809 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.775s user 0.008s sys 0.017s
W20240501 20:01:39.138558 29809 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.777s user 0.009s sys 0.017s
W20240501 20:01:39.136736 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 347273us
W20240501 20:01:39.129285 30753 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.729s user 0.011s sys 0.005s
I20240501 20:01:39.133153 31150 leader_election.cc:288] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
W20240501 20:01:39.134815 30906 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.654s user 0.009s sys 0.024s
I20240501 20:01:39.137102 31087 raft_consensus.cc:2781] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240501 20:01:39.132666 31151 raft_consensus.cc:504] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:39.154299 31017 leader_election.cc:288] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:39.155455 31151 leader_election.cc:288] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:39.157112 30912 leader_election.cc:288] T 36057b3f24d44b9aac3a1378053a395f P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:39.157680 31024 tablet_copy_client.cc:523] T 77be7101c0fe4085b6706296f609870a P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:01:39.165473 31087 raft_consensus.cc:482] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:39.165927 31087 raft_consensus.cc:3037] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:39.168855 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3f29f0eeea4b119984634202e5d9cd" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:39.143821 31098 consensus_meta.cc:287] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d: Time spent flushing consensus metadata: real 0.571s user 0.007s sys 0.003s
I20240501 20:01:39.171535 30935 consensus_queue.cc:260] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:39.180616 29735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 867) took 1382 ms. Trace:
I20240501 20:01:39.180967 29735 rpcz_store.cc:276] 0501 20:01:37.798383 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:37.798510 (+ 127us) service_pool.cc:225] Handling call
0501 20:01:39.180400 (+1381890us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b54000a8054. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787f4fe 00007fe45c7bd9d7 00007fe455ea1c14 00007fe455ea1ba2 00007fe455ea1ad1 00007fe455ea1a4c 00007fe455ea0d02 00007fe455ca2bcc 00007fe455ca2006 00007fe455ca19af
0501 20:01:39.180541 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2606781184}
I20240501 20:01:39.188201 31091 leader_election.cc:288] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:39.172205 29805 raft_consensus.cc:2396] T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240501 20:01:39.173358 30849 consensus_queue.cc:260] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:39.174865 31043 leader_election.cc:288] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:39.204180 29733 raft_consensus.cc:2454] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 3.
I20240501 20:01:39.205386 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5345f6eb62449ba69b4a9c2e349b31" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:39.206219 29804 raft_consensus.cc:2454] T 3c5345f6eb62449ba69b4a9c2e349b31 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 2.
I20240501 20:01:39.208128 29700 leader_election.cc:302] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:39.210304 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e9f75387564275a60d38e9a910319c" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:39.215775 29772 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 354735us
I20240501 20:01:39.185971 31020 raft_consensus.cc:504] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:39.212313 31043 raft_consensus.cc:2781] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:39.214988 30815 leader_election.cc:288] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:39.224300 31017 raft_consensus.cc:482] T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:39.257783 29656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59142 (request call id 878) took 1583 ms. Trace:
I20240501 20:01:39.250198 31152 raft_consensus.cc:482] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:39.291313 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5525fc4b82a44b0991c3dd6098fde81" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:39.291977 29803 raft_consensus.cc:2454] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 3.
I20240501 20:01:39.251832 31019 raft_consensus.cc:482] T 48630ae6eb394ac1b54955dd01cf0ed2 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:39.251827 31043 raft_consensus.cc:482] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:39.295493 29657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37936 (request call id 75) took 1568 ms. Trace:
I20240501 20:01:39.295969 29657 rpcz_store.cc:276] 0501 20:01:37.727582 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:38.871764 (+1144182us) service_pool.cc:225] Handling call
0501 20:01:38.877385 (+ 5621us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:38.943228 (+ 65843us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:38.943257 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:38.943295 (+ 38us) raft_consensus.cc:1631] Marking committed up to 6
0501 20:01:38.943322 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:39.295233 (+351911us) spinlock_profiling.cc:243] Waited 65.7 ms on lock 0x7b54003958d4. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787c1a1 00007fe45787abee 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc 00007fe455e9e782 00007fe455ca2bcc 00007fe455ca2006
0501 20:01:39.295287 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:39.295440 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":151001856}
I20240501 20:01:39.262631 29656 rpcz_store.cc:276] 0501 20:01:37.673954 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:37.705184 (+ 31230us) service_pool.cc:225] Handling call
0501 20:01:37.705530 (+ 346us) raft_consensus.cc:1457] Updating replica for 1 ops
0501 20:01:37.755568 (+ 50038us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:37.755603 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0501 20:01:37.760582 (+ 4979us) log.cc:842] Serialized 26 byte log entry
0501 20:01:37.916907 (+156325us) raft_consensus.cc:1631] Marking committed up to 2
0501 20:01:39.029225 (+1112318us) log.cc:842] Serialized 14 byte log entry
0501 20:01:39.257468 (+228243us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:39.257555 (+ 87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0501 20:01:39.257594 (+ 39us) raft_consensus.cc:1671] finished
0501 20:01:39.257613 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:39.257736 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":306268,"prepare.queue_time_us":151402,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":284757,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":3175,"thread_start_us":1193757,"threads_started":2,"wal-append.queue_time_us":256561,"wal-append.run_cpu_time_us":1570,"wal-append.run_wall_time_us":1010998}
I20240501 20:01:39.333467 29700 leader_election.cc:302] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:39.346215 30417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add9d7579351483fb268f89a44e95d8a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c" is_pre_election: true
I20240501 20:01:39.346865 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e49dd5180f04313ab33942fb3ddc147" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:39.357810 29803 raft_consensus.cc:2396] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f for term 5 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240501 20:01:39.261039 31017 raft_consensus.cc:504] T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:39.333700 31060 raft_consensus.cc:482] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:39.350412 29651 raft_consensus.cc:2454] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 888d99522e4844118f8f1cf26f268c8c in term 6.
I20240501 20:01:39.350876 29802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41864 (request call id 967) took 1100 ms. Trace:
I20240501 20:01:39.352479 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b337b8e3e38d4ad6b1076fbd0d0a1609" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:39.353578 31021 raft_consensus.cc:482] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:39.354022 31155 raft_consensus.cc:482] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:39.354363 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9472c53f79ec423f82c396bd980674e2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:39.354454 31053 raft_consensus.cc:482] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:39.354878 30650 raft_consensus.cc:482] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:39.354629 31118 raft_consensus.cc:504] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } }
I20240501 20:01:39.351810 31083 raft_consensus.cc:482] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:39.355366 31019 raft_consensus.cc:504] T 48630ae6eb394ac1b54955dd01cf0ed2 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:39.256945 31091 raft_consensus.cc:482] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:39.256520 31101 leader_election.cc:288] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:39.255007 31040 raft_consensus.cc:482] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 888d99522e4844118f8f1cf26f268c8c)
I20240501 20:01:39.356037 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e38a2e91a3d48d09a22abce723af833" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:39.259380 31151 raft_consensus.cc:482] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:39.288669 31105 raft_consensus.cc:482] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240501 20:01:39.357250 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 62480.8us
I20240501 20:01:39.286453 30753 raft_consensus.cc:482] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:39.333453 31052 leader_election.cc:288] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:39.410305 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9046e38fc57f4364aa3d539e1fc01928" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:39.425051 31019 leader_election.cc:288] T 48630ae6eb394ac1b54955dd01cf0ed2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:39.425438 31024 tablet_bootstrap.cc:492] T 77be7101c0fe4085b6706296f609870a P 888d99522e4844118f8f1cf26f268c8c: Bootstrap starting.
I20240501 20:01:39.425659 31019 raft_consensus.cc:482] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:39.426183 31019 raft_consensus.cc:504] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:39.424249 29726 raft_consensus.cc:2454] T 9046e38fc57f4364aa3d539e1fc01928 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 1.
I20240501 20:01:39.371032 31107 raft_consensus.cc:482] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:39.366604 31081 raft_consensus.cc:686] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:39.433456 29806 raft_consensus.cc:1261] T 36057b3f24d44b9aac3a1378053a395f P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240501 20:01:39.373889 29698 leader_election.cc:302] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:39.334857 31043 raft_consensus.cc:3037] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:39.356097 31153 raft_consensus.cc:482] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
W20240501 20:01:39.389015 31162 log.cc:924] Time spent T 010ceabf3e5a4b1c9330b80dade765bb P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.099s user 0.000s sys 0.000s
I20240501 20:01:39.421890 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c651044a305c4168805825a53d910899" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:39.440181 29805 raft_consensus.cc:2379] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 3: Already voted for candidate ee0d97247d394a8facfa774f15f3a67d in this term.
I20240501 20:01:39.441417 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be383be783234855878cb8fd442c7b05" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:39.442188 29806 raft_consensus.cc:2379] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 3: Already voted for candidate ee0d97247d394a8facfa774f15f3a67d in this term.
I20240501 20:01:39.442512 29655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59136 (request call id 750) took 1274 ms. Trace:
I20240501 20:01:39.409765 29659 raft_consensus.cc:3037] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:39.408818 31140 raft_consensus.cc:482] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:39.389364 31062 raft_consensus.cc:482] T 2d86f8848a184a87947fd927c44e5a35 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:39.431612 29801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "456035a1cb3f4f7da52d0fc1dd1c656c" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:39.447193 29801 raft_consensus.cc:2454] T 456035a1cb3f4f7da52d0fc1dd1c656c P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 2.
I20240501 20:01:39.365187 31063 raft_consensus.cc:482] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:39.408408 31165 raft_consensus.cc:482] T 1ca3948261374cef8b1b206a43c00173 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:39.454979 31165 raft_consensus.cc:504] T 1ca3948261374cef8b1b206a43c00173 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:39.365109 29802 rpcz_store.cc:276] 0501 20:01:38.250747 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:38.255388 (+ 4641us) service_pool.cc:225] Handling call
0501 20:01:38.268731 (+ 13343us) raft_consensus.cc:1457] Updating replica for 1 ops
0501 20:01:38.269099 (+ 368us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:38.269136 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0501 20:01:38.331729 (+ 62593us) log.cc:842] Serialized 26 byte log entry
0501 20:01:39.135787 (+804058us) raft_consensus.cc:1631] Marking committed up to 1
0501 20:01:39.140274 (+ 4487us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:39.144865 (+ 4591us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0501 20:01:39.350654 (+205789us) raft_consensus.cc:1671] finished
0501 20:01:39.350683 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:39.350824 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3725,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":6629,"thread_start_us":771398,"threads_started":1,"wal-append.queue_time_us":910748}
I20240501 20:01:39.457005 31165 leader_election.cc:288] T 1ca3948261374cef8b1b206a43c00173 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:39.433879 29657 raft_consensus.cc:1226] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20240501 20:01:39.457475 31081 consensus_queue.cc:237] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:39.458302 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8cd40cb4044d18a86001ce6ddf97ab" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:39.459419 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5525fc4b82a44b0991c3dd6098fde81" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:39.459648 31140 raft_consensus.cc:504] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:39.460803 31170 raft_consensus.cc:482] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:39.460966 29653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59136 (request call id 743) took 1399 ms. Trace:
I20240501 20:01:39.461339 31170 raft_consensus.cc:504] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:39.461375 29653 rpcz_store.cc:276] 0501 20:01:38.046453 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:38.330218 (+283765us) service_pool.cc:225] Handling call
0501 20:01:39.445157 (+1114939us) spinlock_profiling.cc:243] Waited 21.3 ms on lock 0x7b5400072bd4. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787f4fe 00007fe45c7bd9d7 00007fe455ea1c14 00007fe455ea1ba2 00007fe455ea1ad1 00007fe455ea1a4c 00007fe455ea0d02 00007fe455ca2bcc 00007fe455ca2006 00007fe455ca19af
0501 20:01:39.445244 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":49077632}
I20240501 20:01:39.422139 31040 raft_consensus.cc:504] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:39.462473 31063 raft_consensus.cc:504] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:39.462908 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfac893b4b4f457694a4eb184b6abb53" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:39.463487 29653 raft_consensus.cc:3037] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:39.463585 31170 leader_election.cc:288] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:39.464097 31040 leader_election.cc:288] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:39.435226 31107 raft_consensus.cc:504] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:39.464459 31063 leader_election.cc:288] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:39.461352 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac2171683884f6f97cdfeb0a656cb49" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:39.465485 29657 raft_consensus.cc:3037] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:39.465579 31181 raft_consensus.cc:482] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:39.465835 31107 leader_election.cc:288] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:39.466094 31181 raft_consensus.cc:504] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:39.466935 31175 raft_consensus.cc:2726] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20240501 20:01:39.467111 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 93352us
I20240501 20:01:39.419359 31091 raft_consensus.cc:504] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:39.422205 31151 raft_consensus.cc:504] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:39.414244 31083 raft_consensus.cc:504] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:39.469110 31091 leader_election.cc:288] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:39.433666 31171 raft_consensus.cc:482] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:39.417462 31169 raft_consensus.cc:482] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:39.408726 31021 raft_consensus.cc:504] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:39.442853 29655 rpcz_store.cc:276] 0501 20:01:38.168613 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:38.921758 (+753145us) service_pool.cc:225] Handling call
0501 20:01:38.922127 (+ 369us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:38.925973 (+ 3846us) raft_consensus.cc:1520] Early marking committed up to index 2
0501 20:01:38.938594 (+ 12621us) log.cc:842] Serialized 14 byte log entry
0501 20:01:39.349578 (+410984us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:39.349641 (+ 63us) raft_consensus.cc:1631] Marking committed up to 2
0501 20:01:39.349695 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:39.442301 (+ 92606us) spinlock_profiling.cc:243] Waited 3.64 ms on lock 0x7b54001c2d54. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787c1a1 00007fe45787abee 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc 00007fe455e9e782 00007fe455ca2bcc 00007fe455ca2006
0501 20:01:39.442345 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:39.442445 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":125634,"prepare.queue_time_us":12438,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":126029,"raft.run_cpu_time_us":82,"raft.run_wall_time_us":82,"spinlock_wait_cycles":9929984,"thread_start_us":284270,"threads_started":1,"wal-append.queue_time_us":489694}
I20240501 20:01:39.443950 31153 raft_consensus.cc:504] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:39.454077 31062 raft_consensus.cc:504] T 2d86f8848a184a87947fd927c44e5a35 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:39.461776 31140 leader_election.cc:288] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:39.467900 29698 leader_election.cc:302] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:39.422859 31101 raft_consensus.cc:482] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 888d99522e4844118f8f1cf26f268c8c)
I20240501 20:01:39.417323 31174 raft_consensus.cc:2781] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:39.294399 31152 raft_consensus.cc:504] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:39.388505 31159 raft_consensus.cc:482] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:39.428817 30753 raft_consensus.cc:504] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:39.421913 31098 raft_consensus.cc:482] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 888d99522e4844118f8f1cf26f268c8c)
I20240501 20:01:39.412580 31053 raft_consensus.cc:504] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:39.389374 31060 raft_consensus.cc:504] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:39.414464 30650 raft_consensus.cc:504] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:39.422266 31105 raft_consensus.cc:504] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:39.472584 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec073fe645614fe28e7837de3c22510b" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:39.824698 29625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 408250us
I20240501 20:01:39.608422 29659 raft_consensus.cc:2454] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 3.
I20240501 20:01:39.470387 29652 raft_consensus.cc:2454] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 3.
I20240501 20:01:39.869972 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6797d82dda2948d8a7445fcd33e44752" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:39.871543 29732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 868) took 2032 ms. Trace:
I20240501 20:01:39.873275 29772 leader_election.cc:302] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
W20240501 20:01:39.875629 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 407782us
I20240501 20:01:39.877167 29698 leader_election.cc:302] T 010ceabf3e5a4b1c9330b80dade765bb P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:39.551878 29657 raft_consensus.cc:2454] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 4.
I20240501 20:01:39.881618 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:39.530915 29653 raft_consensus.cc:2454] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 4.
I20240501 20:01:39.883460 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36057b3f24d44b9aac3a1378053a395f" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:39.879067 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add9d7579351483fb268f89a44e95d8a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:39.883517 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcf30cf66714efdac997862dafbe4a2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:39.438773 31150 raft_consensus.cc:482] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:39.872066 29732 rpcz_store.cc:276] 0501 20:01:37.798726 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:37.799478 (+ 752us) service_pool.cc:225] Handling call
0501 20:01:37.799827 (+ 349us) raft_consensus.cc:1457] Updating replica for 1 ops
0501 20:01:37.876437 (+ 76610us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:37.876484 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0501 20:01:38.222116 (+345632us) spinlock_profiling.cc:243] Waited 11.8 ms on lock 0x7b2c00000a78. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe45625df11 00007fe45626d542 00007fe4578b6fc1 00007fe45787b7da 00007fe45787abee 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc 00007fe455e9e782
0501 20:01:38.222501 (+ 385us) log.cc:842] Serialized 26 byte log entry
0501 20:01:39.105725 (+883224us) raft_consensus.cc:1631] Marking committed up to 2
0501 20:01:39.407526 (+301801us) log.cc:842] Serialized 14 byte log entry
0501 20:01:39.814564 (+407038us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:39.814663 (+ 99us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0501 20:01:39.814744 (+ 81us) raft_consensus.cc:1671] finished
0501 20:01:39.830843 (+ 16099us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:39.831035 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":76025,"prepare.queue_time_us":305212,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":30,"raft.queue_time_us":164668,"raft.run_cpu_time_us":252,"raft.run_wall_time_us":6350,"spinlock_wait_cycles":27137408,"thread_start_us":1539847,"threads_started":3,"wal-append.queue_time_us":981727}
I20240501 20:01:39.435974 31172 raft_consensus.cc:482] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:39.903506 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8edcd903224d48f889b46e1b5318e0ec" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:39.904421 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48fd0a8e289401aa7417c42ff2c81b2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:39.447391 31019 leader_election.cc:288] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
W20240501 20:01:39.907110 29773 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.540s user 0.007s sys 0.007s
W20240501 20:01:39.957888 29773 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.591s user 0.011s sys 0.010s
W20240501 20:01:39.913283 29807 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.502s user 0.016s sys 0.026s
W20240501 20:01:39.958998 29807 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.547s user 0.021s sys 0.031s
I20240501 20:01:39.959533 29659 raft_consensus.cc:1226] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 2. Current term is 3. Ops: []
I20240501 20:01:39.831480 31107 raft_consensus.cc:482] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:39.411648 31155 raft_consensus.cc:504] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:39.963685 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:39.963909 31107 raft_consensus.cc:504] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:39.964468 29659 raft_consensus.cc:1226] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 2. Current term is 3. Ops: []
W20240501 20:01:39.549304 29654 consensus_meta.cc:287] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f: Time spent flushing consensus metadata: real 0.626s user 0.006s sys 0.005s
I20240501 20:01:39.966431 29653 raft_consensus.cc:2454] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 4.
I20240501 20:01:39.967288 29657 raft_consensus.cc:1261] T 2d86f8848a184a87947fd927c44e5a35 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 20:01:39.867030 31174 raft_consensus.cc:482] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:39.532943 31170 raft_consensus.cc:482] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:39.968645 29801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec073fe645614fe28e7837de3c22510b" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:39.513551 31017 leader_election.cc:288] T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:39.513981 31020 leader_election.cc:288] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:39.969826 29809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41864 (request call id 970) took 1618 ms. Trace:
I20240501 20:01:39.679251 31178 raft_consensus.cc:482] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:39.974673 29809 rpcz_store.cc:276] 0501 20:01:38.351253 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:38.351432 (+ 179us) service_pool.cc:225] Handling call
0501 20:01:38.351810 (+ 378us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:38.352076 (+ 266us) raft_consensus.cc:1520] Early marking committed up to index 3
0501 20:01:38.361137 (+ 9061us) log.cc:842] Serialized 14 byte log entry
0501 20:01:39.545597 (+1184460us) log.cc:842] Serialized 14 byte log entry
0501 20:01:39.969431 (+423834us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:39.969482 (+ 51us) raft_consensus.cc:1631] Marking committed up to 3
0501 20:01:39.969539 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:39.969622 (+ 83us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:39.969756 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3896,"prepare.queue_time_us":9320,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":1697,"raft.queue_time_us":116143,"raft.run_cpu_time_us":82,"raft.run_wall_time_us":82,"thread_start_us":1603398,"threads_started":3}
I20240501 20:01:39.986351 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e89c0572e4ee7be8e5315b459bdb3" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:39.986950 29809 raft_consensus.cc:3037] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:39.991595 29659 raft_consensus.cc:1261] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 20:01:39.958864 31091 raft_consensus.cc:2726] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240501 20:01:39.907100 31083 leader_election.cc:288] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:39.867487 31043 raft_consensus.cc:504] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:39.965445 31165 raft_consensus.cc:482] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:40.001008 29660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59142 (request call id 883) took 2311 ms. Trace:
I20240501 20:01:39.973248 31174 raft_consensus.cc:3037] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:39.993002 31020 raft_consensus.cc:482] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:39.968647 31172 raft_consensus.cc:504] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:40.008885 31105 leader_election.cc:288] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:39.903352 31098 raft_consensus.cc:504] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:39.982244 31017 raft_consensus.cc:482] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
W20240501 20:01:39.993546 31168 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.527s user 0.014s sys 0.012s
W20240501 20:01:40.030498 29699 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.615s user 0.012s sys 0.018s
W20240501 20:01:40.031184 29699 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.616s user 0.013s sys 0.018s
I20240501 20:01:39.968966 29654 raft_consensus.cc:2454] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 4.
I20240501 20:01:39.676340 31087 raft_consensus.cc:504] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:40.013312 29660 rpcz_store.cc:276] 0501 20:01:37.679770 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:37.883812 (+204042us) service_pool.cc:225] Handling call
0501 20:01:37.884200 (+ 388us) raft_consensus.cc:1457] Updating replica for 1 ops
0501 20:01:38.049056 (+164856us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:38.049099 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0501 20:01:38.049586 (+ 487us) log.cc:842] Serialized 26 byte log entry
0501 20:01:38.992245 (+942659us) raft_consensus.cc:1631] Marking committed up to 2
0501 20:01:39.003449 (+ 11204us) log.cc:842] Serialized 14 byte log entry
0501 20:01:39.253133 (+249684us) spinlock_profiling.cc:243] Waited 3.92 ms on lock 0x7b18003c8da0. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe45d7f2661 00007fe45d7ff3e1 00007fe4548ca2ea 00007fe45dfb9fe5 00007fe45494e3d4 00007fe45494dcc5 00007fe457873620 00007fe4578857c6 00007fe457896327 00007fe4578962a8 00007fe457896228
0501 20:01:39.257390 (+ 4257us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:39.991096 (+733706us) spinlock_profiling.cc:243] Waited 164 ms on lock 0x7b54001a31d4. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787c1a1 00007fe45787abee 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc 00007fe455e9e782 00007fe455ca2bcc 00007fe455ca2006
0501 20:01:39.991175 (+ 79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0501 20:01:39.991254 (+ 79us) raft_consensus.cc:1671] finished
0501 20:01:39.991276 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:39.991445 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6054,"prepare.queue_time_us":11519,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":51,"raft.queue_time_us":279961,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":17441,"spinlock_wait_cycles":399275008,"thread_start_us":918481,"threads_started":1,"wal-append.queue_time_us":1061712}
I20240501 20:01:39.880066 31101 raft_consensus.cc:504] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:40.020915 30753 leader_election.cc:288] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:39.959079 31150 raft_consensus.cc:504] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:40.025359 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 148248us
I20240501 20:01:39.903200 31169 raft_consensus.cc:504] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:39.902165 31159 raft_consensus.cc:504] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:39.880888 31171 raft_consensus.cc:504] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:40.031234 29809 raft_consensus.cc:2454] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 888d99522e4844118f8f1cf26f268c8c in term 6.
I20240501 20:01:40.010407 29801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9046e38fc57f4364aa3d539e1fc01928" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:39.975747 31170 raft_consensus.cc:504] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:40.037760 31168 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.571s user 0.015s sys 0.013s
I20240501 20:01:40.083915 29801 raft_consensus.cc:2379] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 888d99522e4844118f8f1cf26f268c8c in current term 3: Already voted for candidate ee0d97247d394a8facfa774f15f3a67d in this term.
W20240501 20:01:39.999172 29773 leader_election.cc:334] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 1.568s (SENT)
W20240501 20:01:40.096231 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 97281.5us
I20240501 20:01:40.005671 31186 raft_consensus.cc:2726] T 010ceabf3e5a4b1c9330b80dade765bb P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:01:40.088433 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 428901us
I20240501 20:01:40.035924 31194 raft_consensus.cc:2781] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:01:40.050001 31118 leader_election.cc:288] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:40.017475 31165 raft_consensus.cc:504] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:40.080817 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "696121417ea9403990456cfc6fc4f416" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:40.063520 31155 leader_election.cc:288] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:40.115052 29656 raft_consensus.cc:2454] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 3.
W20240501 20:01:40.106411 29625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 235148us
W20240501 20:01:40.105163 29772 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 231958us
I20240501 20:01:40.110412 29809 raft_consensus.cc:2454] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 888d99522e4844118f8f1cf26f268c8c in term 3.
I20240501 20:01:40.076670 29654 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59136 (request call id 751) took 1799 ms. Trace:
I20240501 20:01:40.027834 31020 raft_consensus.cc:504] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:40.150576 31193 log.cc:924] Time spent T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20240501 20:01:40.150631 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c651044a305c4168805825a53d910899" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:40.151382 29660 raft_consensus.cc:3037] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:40.151772 31020 leader_election.cc:288] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:40.048393 31017 raft_consensus.cc:504] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:39.990293 31178 raft_consensus.cc:504] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:40.098273 30650 leader_election.cc:288] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:40.157977 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 45066.9us
I20240501 20:01:40.106396 31107 leader_election.cc:288] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:40.174876 29772 leader_election.cc:302] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:40.177917 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48630ae6eb394ac1b54955dd01cf0ed2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:40.179805 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3f29f0eeea4b119984634202e5d9cd" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:40.180507 29655 raft_consensus.cc:2454] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 3.
W20240501 20:01:40.154599 31182 log.cc:924] Time spent T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.143s user 0.000s sys 0.001s
I20240501 20:01:40.184058 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211e27a3b0084b89a6dd7550849b5a65" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:40.185441 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69915dd5992641e3a24f7f45f1cd5c0c" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:40.186106 29659 raft_consensus.cc:3037] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:40.185570 31159 leader_election.cc:288] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:40.148984 31194 raft_consensus.cc:686] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [term 3 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:40.152761 31060 leader_election.cc:288] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
W20240501 20:01:40.194744 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 97810.1us
W20240501 20:01:40.195318 29773 leader_election.cc:334] T 84fa28fb55f44b61ba1a88ed775a78ba P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 1.624s (SENT)
I20240501 20:01:40.195247 29801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48630ae6eb394ac1b54955dd01cf0ed2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:40.154659 31132 tablet_copy_client.cc:308] T be383be783234855878cb8fd442c7b05 P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Beginning tablet copy session from remote peer at address 127.27.217.194:32907
I20240501 20:01:40.147234 31091 consensus_queue.cc:1035] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.066s
I20240501 20:01:40.204746 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca3948261374cef8b1b206a43c00173" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:40.205821 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8cd40cb4044d18a86001ce6ddf97ab" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:40.206938 29772 leader_election.cc:302] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:40.198832 29660 raft_consensus.cc:2454] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 3.
I20240501 20:01:40.195801 29773 leader_election.cc:302] T 84fa28fb55f44b61ba1a88ed775a78ba P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:40.148465 29654 rpcz_store.cc:276] 0501 20:01:38.263348 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:38.830288 (+566940us) service_pool.cc:225] Handling call
0501 20:01:40.062639 (+1232351us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:40.197002 31021 leader_election.cc:288] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:40.214620 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 135440us
I20240501 20:01:40.188179 31202 raft_consensus.cc:482] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:40.216578 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38cbeb0dc0c14e3d9182101eea885625" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:40.193050 31020 raft_consensus.cc:482] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:40.218811 31118 raft_consensus.cc:2781] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader election won for term 3
W20240501 20:01:40.221223 31196 log.cc:924] Time spent T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20240501 20:01:40.221577 29659 raft_consensus.cc:2454] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 4.
I20240501 20:01:40.219549 29660 raft_consensus.cc:2454] T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 3.
I20240501 20:01:40.225072 29698 leader_election.cc:302] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:40.197706 31062 leader_election.cc:288] T 2d86f8848a184a87947fd927c44e5a35 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:40.226938 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f5d1686214485d91a0baf4b870740e" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:40.227640 29655 raft_consensus.cc:2379] T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 4: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:40.227895 31098 leader_election.cc:288] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:40.230417 31098 raft_consensus.cc:2726] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:40.221653 31175 consensus_queue.cc:1046] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20240501 20:01:40.232249 31192 log.cc:924] Time spent T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.078s user 0.001s sys 0.000s
I20240501 20:01:40.218530 31060 raft_consensus.cc:2781] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:01:40.236888 29625 leader_election.cc:302] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:40.260486 31194 consensus_queue.cc:237] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:40.233728 31198 consensus_queue.cc:1035] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20240501 20:01:40.226567 31043 leader_election.cc:288] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:40.236778 31087 leader_election.cc:288] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 5 election: Requested vote from peers ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:40.226214 31118 raft_consensus.cc:686] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [term 3 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:40.235692 29772 leader_election.cc:302] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, 888d99522e4844118f8f1cf26f268c8c
I20240501 20:01:40.270339 29772 leader_election.cc:302] T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:40.270787 31209 raft_consensus.cc:2726] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:01:40.271100 31206 raft_consensus.cc:2781] T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:40.271436 31206 raft_consensus.cc:482] T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:40.271749 31206 raft_consensus.cc:3037] T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:40.272908 29772 leader_election.cc:302] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:40.273803 29772 leader_election.cc:302] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:40.259778 31172 leader_election.cc:288] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:40.253247 31186 raft_consensus.cc:3032] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Stepping down as leader of term 4
I20240501 20:01:40.277137 31209 raft_consensus.cc:2781] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:40.284158 29807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41864 (request call id 978) took 1354 ms. Trace:
I20240501 20:01:40.284597 31209 raft_consensus.cc:482] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:40.285283 31209 raft_consensus.cc:3037] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:40.268325 30650 raft_consensus.cc:2781] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:40.284622 29807 rpcz_store.cc:276] 0501 20:01:38.929800 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:38.929969 (+ 169us) service_pool.cc:225] Handling call
0501 20:01:38.930345 (+ 376us) raft_consensus.cc:1457] Updating replica for 1 ops
0501 20:01:39.407821 (+477476us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:39.407859 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0501 20:01:39.411264 (+ 3405us) log.cc:842] Serialized 26 byte log entry
0501 20:01:39.960189 (+548925us) raft_consensus.cc:1631] Marking committed up to 1
0501 20:01:39.960241 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:39.960330 (+ 89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0501 20:01:40.283954 (+323624us) raft_consensus.cc:1671] finished
0501 20:01:40.283974 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:40.284107 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":90909,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":9028,"thread_start_us":1025596,"threads_started":2,"wal-append.queue_time_us":697711}
I20240501 20:01:40.289073 30650 raft_consensus.cc:482] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:40.289597 30650 raft_consensus.cc:3037] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:40.277482 31170 leader_election.cc:288] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:40.292624 31170 raft_consensus.cc:2781] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:01:40.293188 31170 raft_consensus.cc:686] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [term 3 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:40.287496 31017 leader_election.cc:288] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
W20240501 20:01:40.288612 31149 log.cc:924] Time spent T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 1.251s user 0.000s sys 0.011s
W20240501 20:01:40.297573 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 72537.3us
I20240501 20:01:40.264415 31060 raft_consensus.cc:686] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [term 4 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:40.260113 31171 leader_election.cc:288] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:40.289541 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d2cac25f354a97b1bfb53d57ae7c08" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:40.302955 29807 raft_consensus.cc:2454] T d1d2cac25f354a97b1bfb53d57ae7c08 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 3.
I20240501 20:01:40.304076 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f5d1686214485d91a0baf4b870740e" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:40.304785 29807 raft_consensus.cc:2454] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 3.
I20240501 20:01:40.224745 31020 raft_consensus.cc:504] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:40.233961 31135 raft_consensus.cc:2726] T 84fa28fb55f44b61ba1a88ed775a78ba P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 20:01:40.342348 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 117295us
I20240501 20:01:40.298838 29698 leader_election.cc:302] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:40.274055 31101 leader_election.cc:288] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:40.308743 29807 raft_consensus.cc:1261] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20240501 20:01:40.418448 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 75189.3us
I20240501 20:01:40.233963 31169 leader_election.cc:288] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
W20240501 20:01:40.417819 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 119429us
W20240501 20:01:40.419009 29773 leader_election.cc:334] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 2.089s (SENT)
W20240501 20:01:40.418121 29772 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 144381us
I20240501 20:01:40.287060 31186 raft_consensus.cc:729] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:40.422520 29772 leader_election.cc:302] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:40.422816 29698 leader_election.cc:302] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:40.421975 29773 leader_election.cc:302] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:40.426919 29772 leader_election.cc:302] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:40.430233 29772 leader_election.cc:302] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:40.431994 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f261d2a05bf4c838532049601694b4a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:40.434876 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77be7101c0fe4085b6706296f609870a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:40.294288 31170 consensus_queue.cc:237] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:40.435292 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e49dd5180f04313ab33942fb3ddc147" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:40.418820 31043 raft_consensus.cc:2726] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 20:01:40.224834 31202 raft_consensus.cc:504] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:40.421298 31159 raft_consensus.cc:482] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:40.421922 31107 raft_consensus.cc:482] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:40.438735 29659 raft_consensus.cc:2454] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 4.
I20240501 20:01:40.444126 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c597f15e844f30837012baed4b0501" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:40.446635 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0374bb790ed345ed90452595f71383a2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:40.446930 29655 raft_consensus.cc:2454] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 4.
I20240501 20:01:40.438802 29698 leader_election.cc:302] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:40.445585 31106 raft_consensus.cc:482] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:40.453073 31106 raft_consensus.cc:504] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:40.454982 31106 leader_election.cc:288] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:40.455629 29654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48fd0a8e289401aa7417c42ff2c81b2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:40.434636 31169 raft_consensus.cc:2781] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:01:40.460309 31214 raft_consensus.cc:2781] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:40.433212 31017 raft_consensus.cc:2726] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 20:01:40.436574 31135 raft_consensus.cc:2781] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:01:40.475598 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3f29f0eeea4b119984634202e5d9cd" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:40.446130 30385 leader_election.cc:302] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 888d99522e4844118f8f1cf26f268c8c, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:40.439954 31201 raft_consensus.cc:2726] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 20:01:40.498965 31135 raft_consensus.cc:686] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [term 4 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
W20240501 20:01:40.499223 30385 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 61121.5us
I20240501 20:01:40.458374 31159 raft_consensus.cc:504] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:40.448180 31194 raft_consensus.cc:482] T 010ceabf3e5a4b1c9330b80dade765bb P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:40.457937 31042 leader_election.cc:288] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:40.457753 31206 raft_consensus.cc:504] T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:40.466718 31214 raft_consensus.cc:482] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:40.528420 31202 leader_election.cc:288] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:40.458549 31043 raft_consensus.cc:482] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:40.530736 31205 raft_consensus.cc:2726] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240501 20:01:40.531234 31212 raft_consensus.cc:482] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:40.531845 31144 raft_consensus.cc:2781] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 6 FOLLOWER]: Leader election won for term 6
W20240501 20:01:40.500926 31149 log.cc:924] Time spent T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20240501 20:01:40.534405 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9046e38fc57f4364aa3d539e1fc01928" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:40.556741 29733 raft_consensus.cc:3037] T 9046e38fc57f4364aa3d539e1fc01928 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 3
I20240501 20:01:40.543859 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "696121417ea9403990456cfc6fc4f416" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
W20240501 20:01:40.548373 30383 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 96653.6us
W20240501 20:01:40.552276 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 72928.4us
I20240501 20:01:40.462241 31107 raft_consensus.cc:504] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:40.440323 31118 consensus_queue.cc:237] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:40.553007 29807 raft_consensus.cc:2379] T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 4: Already voted for candidate ee0d97247d394a8facfa774f15f3a67d in this term.
I20240501 20:01:40.559746 29698 leader_election.cc:302] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:40.568243 29653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59136 (request call id 759) took 1215 ms. Trace:
I20240501 20:01:40.568265 29735 raft_consensus.cc:1226] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20240501 20:01:40.568608 29653 rpcz_store.cc:276] 0501 20:01:39.353022 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:39.883329 (+530307us) service_pool.cc:225] Handling call
0501 20:01:40.568064 (+684735us) spinlock_profiling.cc:243] Waited 73.1 ms on lock 0x7b54000730d4. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787f4fe 00007fe45c7bd9d7 00007fe455ea1c14 00007fe455ea1ba2 00007fe455ea1ad1 00007fe455ea1a4c 00007fe455ea0d02 00007fe455ca2bcc 00007fe455ca2006 00007fe455ca19af
0501 20:01:40.568198 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":168073472}
I20240501 20:01:40.553089 31043 raft_consensus.cc:504] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:40.547255 31214 raft_consensus.cc:3037] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:40.572470 29745 tablet_copy_service.cc:140] P fdfeac4190e045f290e003b2298d988d: Received BeginTabletCopySession request for tablet be383be783234855878cb8fd442c7b05 from peer 888d99522e4844118f8f1cf26f268c8c ({username='slave'} at 127.0.0.1:47414)
I20240501 20:01:40.572594 31219 raft_consensus.cc:482] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:40.574218 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e89c0572e4ee7be8e5315b459bdb3" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:40.572908 31159 leader_election.cc:288] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:40.465747 31060 consensus_queue.cc:237] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:40.499837 31135 consensus_queue.cc:237] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } }
I20240501 20:01:40.579047 29733 raft_consensus.cc:2454] T 9046e38fc57f4364aa3d539e1fc01928 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 888d99522e4844118f8f1cf26f268c8c in term 3.
I20240501 20:01:40.580010 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d88139f89c47ff84cad674147aae0f" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:40.580572 29732 raft_consensus.cc:3037] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:40.536514 31194 raft_consensus.cc:504] T 010ceabf3e5a4b1c9330b80dade765bb P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:40.549475 30385 leader_election.cc:302] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 888d99522e4844118f8f1cf26f268c8c, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:40.557026 31172 consensus_queue.cc:1046] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20240501 20:01:40.569766 31053 leader_election.cc:288] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:40.592638 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:40.597839 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84fa28fb55f44b61ba1a88ed775a78ba" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:40.599228 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d88139f89c47ff84cad674147aae0f" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:40.606506 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e9f75387564275a60d38e9a910319c" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:40.607777 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5525fc4b82a44b0991c3dd6098fde81" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:40.608964 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5345f6eb62449ba69b4a9c2e349b31" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:40.610309 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd5e83faaff4e2da4e4cc22ba410d33" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:40.579269 31144 raft_consensus.cc:686] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 6 LEADER]: Becoming Leader. State: Replica: 888d99522e4844118f8f1cf26f268c8c, State: Running, Role: LEADER
I20240501 20:01:40.573045 29745 tablet_copy_service.cc:161] P fdfeac4190e045f290e003b2298d988d: Beginning new tablet copy session on tablet be383be783234855878cb8fd442c7b05 from peer 888d99522e4844118f8f1cf26f268c8c at {username='slave'} at 127.0.0.1:47414: session id = 888d99522e4844118f8f1cf26f268c8c-be383be783234855878cb8fd442c7b05
I20240501 20:01:40.614697 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec073fe645614fe28e7837de3c22510b" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:40.614926 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f261d2a05bf4c838532049601694b4a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:40.615499 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a2bba62f434d1c8489b55c8fec2187" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:40.616482 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5345f6eb62449ba69b4a9c2e349b31" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:40.613322 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d2cac25f354a97b1bfb53d57ae7c08" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:40.617643 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f5d1686214485d91a0baf4b870740e" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:40.618302 29655 raft_consensus.cc:2454] T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 4.
I20240501 20:01:40.618017 29653 raft_consensus.cc:2454] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 3.
I20240501 20:01:40.591672 31175 raft_consensus.cc:2726] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240501 20:01:40.586301 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8edcd903224d48f889b46e1b5318e0ec" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:40.621369 29660 raft_consensus.cc:1261] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240501 20:01:40.622049 29653 raft_consensus.cc:1261] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 20:01:40.622260 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3f29f0eeea4b119984634202e5d9cd" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:40.600977 31020 leader_election.cc:288] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:40.625918 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e49dd5180f04313ab33942fb3ddc147" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:40.626696 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49da4246f5f6438abe0e7e8c9a523d80" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:40.623119 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e07220db7444ae97f294d70eda6a93" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:40.623385 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48630ae6eb394ac1b54955dd01cf0ed2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:40.612569 31169 raft_consensus.cc:686] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [term 4 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:40.627367 29727 raft_consensus.cc:2379] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 4: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
W20240501 20:01:40.627761 29727 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 779) took 1618 ms (client timeout 2129 ms). Trace:
I20240501 20:01:40.628028 29659 raft_consensus.cc:1261] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20240501 20:01:40.628436 29659 pending_rounds.cc:76] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f: Aborting all ops after (but not including) 1
W20240501 20:01:40.628131 29727 rpcz_store.cc:269] 0501 20:01:39.009355 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:39.013947 (+ 4592us) service_pool.cc:225] Handling call
0501 20:01:40.578726 (+1564779us) spinlock_profiling.cc:243] Waited 5.23 ms on lock 0x7b58001b3098. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe454797b51 00007fe4549193b1 00007fe454919262 00007fe45d80b889 00007fe45c7483c0 00007fe45c80e3bb 00007fe45c80e441 00007fe45c7bdf60 00007fe45c7bd844 00007fe455ea1c14 00007fe455ea1ba2
0501 20:01:40.627714 (+ 48988us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12040448}
I20240501 20:01:40.628782 29659 pending_rounds.cc:97] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20240501 20:01:40.629019 29659 raft_consensus.cc:2866] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 20:01:40.629355 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be383be783234855878cb8fd442c7b05" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:40.629493 30650 raft_consensus.cc:504] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:40.630079 29727 raft_consensus.cc:2379] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 3: Already voted for candidate ee0d97247d394a8facfa774f15f3a67d in this term.
I20240501 20:01:40.630897 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "456035a1cb3f4f7da52d0fc1dd1c656c" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:40.475863 31186 consensus_queue.cc:260] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:40.630957 31204 raft_consensus.cc:482] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:40.624364 29655 raft_consensus.cc:2454] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 3.
I20240501 20:01:40.623754 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38cbeb0dc0c14e3d9182101eea885625" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:40.635207 29660 raft_consensus.cc:2396] T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20240501 20:01:40.616823 30385 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 83046.2us
I20240501 20:01:40.619725 29658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59142 (request call id 884) took 2935 ms. Trace:
I20240501 20:01:40.627476 30417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05df43bdfa479b902b1da297fb5974" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c" is_pre_election: true
I20240501 20:01:40.632170 31175 raft_consensus.cc:3032] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:01:40.624646 31202 raft_consensus.cc:482] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:40.652009 29700 leader_election.cc:302] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:40.649924 29660 raft_consensus.cc:1226] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 2. Current term is 3. Ops: []
I20240501 20:01:40.652916 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e38a2e91a3d48d09a22abce723af833" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:40.631644 31205 raft_consensus.cc:482] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:40.655436 29700 leader_election.cc:302] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:40.635972 29654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ed27033ab1645e9ab0600ab635d9589" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
W20240501 20:01:40.650362 31213 log.cc:924] Time spent T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d: Append to log took a long time: real 0.114s user 0.000s sys 0.001s
I20240501 20:01:40.573060 31219 raft_consensus.cc:504] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:40.640776 30650 leader_election.cc:288] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:40.654420 31199 consensus_queue.cc:1035] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.069s
I20240501 20:01:40.663884 31042 raft_consensus.cc:2781] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:40.664341 30414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd5e83faaff4e2da4e4cc22ba410d33" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c" is_pre_election: true
I20240501 20:01:40.677449 29772 leader_election.cc:302] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
W20240501 20:01:40.660715 29773 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 1.507s (SENT) [suppressed 21 similar messages]
I20240501 20:01:40.571684 31212 raft_consensus.cc:504] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:40.659134 31043 leader_election.cc:288] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:40.653770 31175 raft_consensus.cc:729] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:40.658643 31202 raft_consensus.cc:504] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:40.681804 29772 leader_election.cc:302] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:40.634665 31204 raft_consensus.cc:504] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:40.648258 31211 raft_consensus.cc:482] T 1ca3948261374cef8b1b206a43c00173 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:40.697510 31135 raft_consensus.cc:482] T a8787bc4d6b44649b9138d906730110a P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:40.669423 31206 leader_election.cc:288] T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
W20240501 20:01:40.688649 31174 consensus_meta.cc:287] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d: Time spent flushing consensus metadata: real 0.659s user 0.010s sys 0.000s
I20240501 20:01:40.635986 31169 consensus_queue.cc:237] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:40.677949 31042 raft_consensus.cc:482] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:40.681406 31106 raft_consensus.cc:2781] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240501 20:01:40.688381 30912 raft_consensus.cc:2781] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240501 20:01:40.705293 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1a08b455b94ae089396a15c837ddc2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:40.694294 31107 leader_election.cc:288] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:40.693966 31143 raft_consensus.cc:2781] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240501 20:01:40.756906 29657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59142 (request call id 896) took 1341 ms. Trace:
I20240501 20:01:40.757409 29657 rpcz_store.cc:276] 0501 20:01:39.415657 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:39.964590 (+548933us) service_pool.cc:225] Handling call
0501 20:01:39.964878 (+ 288us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:39.967712 (+ 2834us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:40.756624 (+788912us) spinlock_profiling.cc:243] Waited 2.03 ms on lock 0x7b54001c1954. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787c1a1 00007fe45787abee 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc 00007fe455e9e782 00007fe455ca2bcc 00007fe455ca2006
0501 20:01:40.756836 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4664576}
I20240501 20:01:40.757323 31219 leader_election.cc:288] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:40.698966 30417 raft_consensus.cc:2454] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 3.
W20240501 20:01:40.760777 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 87913.2us
I20240501 20:01:40.703599 30415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e07220db7444ae97f294d70eda6a93" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c" is_pre_election: true
I20240501 20:01:40.751468 31042 raft_consensus.cc:3037] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:40.751633 31106 raft_consensus.cc:482] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:40.753096 30912 raft_consensus.cc:482] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:40.660614 31205 raft_consensus.cc:504] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:40.762125 31024 log.cc:924] Time spent T 77be7101c0fe4085b6706296f609870a P 888d99522e4844118f8f1cf26f268c8c: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20240501 20:01:40.655014 31144 consensus_queue.cc:237] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:40.777710 31229 raft_consensus.cc:482] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:40.776330 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211e27a3b0084b89a6dd7550849b5a65" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:40.706548 31174 raft_consensus.cc:504] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:40.787833 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be383be783234855878cb8fd442c7b05" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:40.810436 29653 raft_consensus.cc:2454] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 3.
I20240501 20:01:40.768800 31194 leader_election.cc:288] T 010ceabf3e5a4b1c9330b80dade765bb P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:40.803088 29732 raft_consensus.cc:2454] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 3.
I20240501 20:01:40.806677 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38cbeb0dc0c14e3d9182101eea885625" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:40.704308 31135 raft_consensus.cc:504] T a8787bc4d6b44649b9138d906730110a P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:40.759328 31143 raft_consensus.cc:482] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:40.703040 31211 raft_consensus.cc:504] T 1ca3948261374cef8b1b206a43c00173 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:40.780578 31106 raft_consensus.cc:3037] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:40.813700 29730 raft_consensus.cc:1226] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 3. Current term is 4. Ops: []
I20240501 20:01:40.814709 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9046e38fc57f4364aa3d539e1fc01928" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:40.818626 29732 raft_consensus.cc:2379] T 9046e38fc57f4364aa3d539e1fc01928 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 3: Already voted for candidate 888d99522e4844118f8f1cf26f268c8c in this term.
I20240501 20:01:40.782315 30912 raft_consensus.cc:3037] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:40.754010 31175 consensus_queue.cc:260] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:40.819588 31175 raft_consensus.cc:3037] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:40.815754 31174 leader_election.cc:288] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:40.805136 29745 tablet_copy_source_session.cc:205] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:01:40.828162 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3f29f0eeea4b119984634202e5d9cd" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:40.833686 29732 raft_consensus.cc:2454] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 3.
I20240501 20:01:40.840605 31211 leader_election.cc:288] T 1ca3948261374cef8b1b206a43c00173 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:40.807585 31204 leader_election.cc:288] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:40.847040 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211e27a3b0084b89a6dd7550849b5a65" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:40.816929 31143 raft_consensus.cc:3037] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:40.864122 29698 leader_election.cc:302] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:40.864603 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c651044a305c4168805825a53d910899" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:40.865613 29698 leader_election.cc:302] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:40.865787 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca3948261374cef8b1b206a43c00173" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:40.866547 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcf30cf66714efdac997862dafbe4a2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:40.867301 29657 raft_consensus.cc:2396] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240501 20:01:40.867170 29698 leader_election.cc:302] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:40.869778 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5525fc4b82a44b0991c3dd6098fde81" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:40.865170 29732 raft_consensus.cc:2454] T c651044a305c4168805825a53d910899 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 2.
I20240501 20:01:40.828146 31194 raft_consensus.cc:482] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240501 20:01:40.838745 31215 log.cc:924] Time spent T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d: Append to log took a long time: real 0.214s user 0.001s sys 0.000s
I20240501 20:01:40.816280 31223 raft_consensus.cc:2781] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:40.828064 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ed27033ab1645e9ab0600ab635d9589" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
W20240501 20:01:40.846894 31209 consensus_meta.cc:287] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d: Time spent flushing consensus metadata: real 0.561s user 0.010s sys 0.000s
I20240501 20:01:40.878206 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69915dd5992641e3a24f7f45f1cd5c0c" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:40.878728 29654 raft_consensus.cc:1226] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20240501 20:01:40.879005 31202 leader_election.cc:288] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:40.879272 30415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05df43bdfa479b902b1da297fb5974" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c"
I20240501 20:01:40.879139 31214 raft_consensus.cc:504] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:40.893934 31234 raft_consensus.cc:482] T c651044a305c4168805825a53d910899 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:40.894629 29807 raft_consensus.cc:2379] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d in current term 4: Already voted for candidate ee0d97247d394a8facfa774f15f3a67d in this term.
I20240501 20:01:40.888156 29698 leader_election.cc:302] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:40.897464 31233 raft_consensus.cc:2726] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:01:40.898074 31233 raft_consensus.cc:2726] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:40.900365 29698 leader_election.cc:302] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:40.903264 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6797d82dda2948d8a7445fcd33e44752" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:40.904294 29732 raft_consensus.cc:2379] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 4: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:40.904732 29698 leader_election.cc:302] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:40.908676 30418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ed27033ab1645e9ab0600ab635d9589" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c" is_pre_election: true
I20240501 20:01:40.909312 31205 leader_election.cc:288] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:40.891566 31211 raft_consensus.cc:2726] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:40.910966 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c651044a305c4168805825a53d910899" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:40.911962 31205 raft_consensus.cc:2726] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:40.912802 31204 raft_consensus.cc:2726] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:01:40.647392 29658 rpcz_store.cc:276] 0501 20:01:37.682880 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:37.946080 (+263200us) service_pool.cc:225] Handling call
0501 20:01:38.020082 (+ 74002us) raft_consensus.cc:1457] Updating replica for 1 ops
0501 20:01:38.020535 (+ 453us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:38.020576 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0501 20:01:38.021221 (+ 645us) log.cc:842] Serialized 26 byte log entry
0501 20:01:38.845400 (+824179us) raft_consensus.cc:1631] Marking committed up to 2
0501 20:01:38.880292 (+ 34892us) log.cc:842] Serialized 14 byte log entry
0501 20:01:38.880498 (+ 206us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:38.880580 (+ 82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0501 20:01:40.273166 (+1392586us) spinlock_profiling.cc:243] Waited 44 ms on lock 0x7b44000232cc. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe454797b51 00007fe454865685 00007fe4548660c2 00007fe45750e441 00007fe457509d4a 00007fe4574fdb1e 00007fe4574fce66 00007fe4574fca88 00007fe457506481 00007fe457506439 00007fe4575063c9
0501 20:01:40.582940 (+309774us) raft_consensus.cc:1671] finished
0501 20:01:40.612096 (+ 29156us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:40.618627 (+ 6531us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":483,"prepare.queue_time_us":24022,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":8941,"raft.run_cpu_time_us":237,"raft.run_wall_time_us":4963,"spinlock_wait_cycles":101252992,"thread_start_us":812454,"threads_started":1,"wal-append.queue_time_us":973378}
I20240501 20:01:40.916189 29698 leader_election.cc:302] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:40.892154 31223 raft_consensus.cc:482] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:40.935123 31230 raft_consensus.cc:2726] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:40.933059 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9046e38fc57f4364aa3d539e1fc01928" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:40.937976 29732 raft_consensus.cc:2379] T 9046e38fc57f4364aa3d539e1fc01928 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 3: Already voted for candidate 888d99522e4844118f8f1cf26f268c8c in this term.
I20240501 20:01:40.938831 29698 leader_election.cc:302] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:40.881296 31194 raft_consensus.cc:504] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:40.943935 31205 consensus_queue.cc:1035] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:40.892882 31209 raft_consensus.cc:504] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:40.902722 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b337b8e3e38d4ad6b1076fbd0d0a1609" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:40.911468 31181 leader_election.cc:288] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:40.817871 31229 raft_consensus.cc:504] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:40.930025 31202 raft_consensus.cc:482] T 3c5345f6eb62449ba69b4a9c2e349b31 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:40.949155 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b337b8e3e38d4ad6b1076fbd0d0a1609" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:40.936527 31212 leader_election.cc:288] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:40.954173 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1a08b455b94ae089396a15c837ddc2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:40.931326 31223 raft_consensus.cc:3037] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:40.954008 31205 consensus_queue.cc:1046] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20240501 20:01:40.955854 31194 leader_election.cc:288] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:40.942817 31233 raft_consensus.cc:2726] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:01:40.957639 31202 raft_consensus.cc:504] T 3c5345f6eb62449ba69b4a9c2e349b31 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:40.943290 31017 raft_consensus.cc:482] T a8e9f75387564275a60d38e9a910319c P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:40.936223 31211 consensus_queue.cc:1035] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.019s
I20240501 20:01:40.955353 31042 raft_consensus.cc:504] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:40.961673 31106 raft_consensus.cc:504] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:40.961673 31024 tablet_bootstrap.cc:492] T 77be7101c0fe4085b6706296f609870a P 888d99522e4844118f8f1cf26f268c8c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 20:01:40.894484 31234 raft_consensus.cc:504] T c651044a305c4168805825a53d910899 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:40.977206 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e38a2e91a3d48d09a22abce723af833" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:40.980408 29732 raft_consensus.cc:2454] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 3.
I20240501 20:01:40.977720 31220 raft_consensus.cc:2781] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:40.978101 31221 raft_consensus.cc:482] T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:40.979368 29803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41858 (request call id 917) took 1591 ms. Trace:
I20240501 20:01:40.978518 31235 consensus_queue.cc:1035] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.036s
I20240501 20:01:40.982234 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0374bb790ed345ed90452595f71383a2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:40.959981 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8cd40cb4044d18a86001ce6ddf97ab" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:40.984506 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31fff64b453a423eb0f567b3171564fb" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:40.989729 29800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41858 (request call id 928) took 1474 ms. Trace:
I20240501 20:01:40.994648 29807 raft_consensus.cc:2396] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20240501 20:01:40.994402 29800 rpcz_store.cc:276] 0501 20:01:39.514922 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:39.515529 (+ 607us) service_pool.cc:225] Handling call
0501 20:01:40.989069 (+1473540us) spinlock_profiling.cc:243] Waited 10.2 ms on lock 0x7b58000d2c98. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe454797b51 00007fe4549193b1 00007fe454919262 00007fe45d80b889 00007fe45c7483c0 00007fe45c80e3bb 00007fe45c80e441 00007fe45c7d2060 00007fe45c7bd5a9 00007fe45c7bd052 00007fe455e9f694
0501 20:01:40.989241 (+ 172us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:40.989408 (+ 167us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:40.989433 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:40.989487 (+ 54us) raft_consensus.cc:1631] Marking committed up to 3
0501 20:01:40.989523 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:40.989576 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:40.989679 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":23536512}
I20240501 20:01:40.995878 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be383be783234855878cb8fd442c7b05" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:40.982846 29802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49da4246f5f6438abe0e7e8c9a523d80" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:40.990947 29803 rpcz_store.cc:276] 0501 20:01:39.388189 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:39.388374 (+ 185us) service_pool.cc:225] Handling call
0501 20:01:39.417166 (+ 28792us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:39.460301 (+ 43135us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:39.460329 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:39.460369 (+ 40us) raft_consensus.cc:1631] Marking committed up to 1
0501 20:01:39.460398 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:40.979152 (+1518754us) spinlock_profiling.cc:243] Waited 42.9 ms on lock 0x7b540036bd54. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787c1a1 00007fe45787abee 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc 00007fe455e9e782 00007fe455ca2bcc 00007fe455ca2006
0501 20:01:40.979202 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:40.979320 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":98647808}
I20240501 20:01:40.992336 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5525fc4b82a44b0991c3dd6098fde81" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:40.964016 31017 raft_consensus.cc:504] T a8e9f75387564275a60d38e9a910319c P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:40.999100 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3f29f0eeea4b119984634202e5d9cd" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:41.013947 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1a08b455b94ae089396a15c837ddc2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:41.014510 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010ceabf3e5a4b1c9330b80dade765bb" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:41.013947 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1a08b455b94ae089396a15c837ddc2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:41.011945 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcf30cf66714efdac997862dafbe4a2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:41.013084 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07bdb8b09b8c4ef4b8e49daa9b8c01d6" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:41.018797 29807 raft_consensus.cc:2454] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 3.
I20240501 20:01:41.018960 29803 raft_consensus.cc:2454] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 4.
I20240501 20:01:41.017424 29660 raft_consensus.cc:2454] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 3.
I20240501 20:01:41.020385 29654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a2bba62f434d1c8489b55c8fec2187" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:41.020663 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5345f6eb62449ba69b4a9c2e349b31" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:41.021253 29800 raft_consensus.cc:3037] T 3c5345f6eb62449ba69b4a9c2e349b31 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:41.021473 31017 leader_election.cc:288] T a8e9f75387564275a60d38e9a910319c P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:41.022428 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3f29f0eeea4b119984634202e5d9cd" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
W20240501 20:01:41.024504 29802 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41858 (request call id 925) took 1554 ms (client timeout 1722 ms). Trace:
W20240501 20:01:41.024891 29802 rpcz_store.cc:269] 0501 20:01:39.469717 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:39.492287 (+ 22570us) service_pool.cc:225] Handling call
0501 20:01:40.982689 (+1490402us) spinlock_profiling.cc:243] Waited 20.9 ms on lock 0x7b58000d2c98. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe454797b51 00007fe4549193b1 00007fe454919262 00007fe45d80b889 00007fe45c7483c0 00007fe45c80e3bb 00007fe45c80e441 00007fe45c7bdf60 00007fe45c7bd844 00007fe455ea1c14 00007fe455ea1ba2
0501 20:01:41.024431 (+ 41742us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":48044800}
I20240501 20:01:41.026129 29802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "456035a1cb3f4f7da52d0fc1dd1c656c" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
W20240501 20:01:41.026500 29656 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59136 (request call id 757) took 1673 ms (client timeout 1727 ms). Trace:
I20240501 20:01:41.026654 29802 raft_consensus.cc:3037] T 456035a1cb3f4f7da52d0fc1dd1c656c P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Advancing to term 3
W20240501 20:01:41.026978 29656 rpcz_store.cc:269] 0501 20:01:39.352575 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:39.355828 (+ 3253us) service_pool.cc:225] Handling call
0501 20:01:41.026343 (+1670515us) spinlock_profiling.cc:243] Waited 700 ms on lock 0x7b54001a1dd4. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787f4fe 00007fe45c7bd9d7 00007fe455ea1c14 00007fe455ea1ba2 00007fe455ea1ad1 00007fe455ea1a4c 00007fe455ea0d02 00007fe455ca2bcc 00007fe455ca2006 00007fe455ca19af
0501 20:01:41.026446 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1609069312}
I20240501 20:01:41.016953 29653 raft_consensus.cc:3037] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:41.039685 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649b71f9f267437ca51c758dd9ca6987" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:41.028374 31151 leader_election.cc:288] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:41.041697 29804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41858 (request call id 913) took 1707 ms. Trace:
I20240501 20:01:41.012322 31202 leader_election.cc:288] T 3c5345f6eb62449ba69b4a9c2e349b31 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:41.042104 29804 rpcz_store.cc:276] 0501 20:01:39.334618 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:39.334728 (+ 110us) service_pool.cc:225] Handling call
0501 20:01:39.387714 (+ 52986us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:39.421854 (+ 34140us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:39.421883 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:39.421925 (+ 42us) raft_consensus.cc:1631] Marking committed up to 2
0501 20:01:39.421955 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:41.041498 (+1619543us) spinlock_profiling.cc:243] Waited 33.9 ms on lock 0x7b5400210554. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787c1a1 00007fe45787abee 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc 00007fe455e9e782 00007fe455ca2bcc 00007fe455ca2006
0501 20:01:41.041542 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:41.041649 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":78045440}
I20240501 20:01:41.044011 29805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41858 (request call id 926) took 1529 ms. Trace:
I20240501 20:01:41.052361 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05df43bdfa479b902b1da297fb5974" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:41.052789 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d2cac25f354a97b1bfb53d57ae7c08" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:41.046205 29805 rpcz_store.cc:276] 0501 20:01:39.514332 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:39.517766 (+ 3434us) service_pool.cc:225] Handling call
0501 20:01:41.043388 (+1525622us) spinlock_profiling.cc:243] Waited 3.29 ms on lock 0x7b58000d2c98. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe454797b51 00007fe4549193b1 00007fe454919262 00007fe45d80b889 00007fe45c7483c0 00007fe45c80e3bb 00007fe45c80e441 00007fe45c7d2060 00007fe45c7bd5a9 00007fe45c7bd052 00007fe455e9f694
0501 20:01:41.043580 (+ 192us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:41.043738 (+ 158us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:41.043763 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:41.043800 (+ 37us) raft_consensus.cc:1631] Marking committed up to 2
0501 20:01:41.043827 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:41.043875 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:41.043964 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7573888}
I20240501 20:01:41.053090 29804 raft_consensus.cc:3037] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:41.053994 29807 raft_consensus.cc:2396] T d1d2cac25f354a97b1bfb53d57ae7c08 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240501 20:01:41.059520 29800 raft_consensus.cc:2454] T 3c5345f6eb62449ba69b4a9c2e349b31 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 3.
I20240501 20:01:41.060766 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a35893b91247a99912eba787d787cf" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:41.067576 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c597f15e844f30837012baed4b0501" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:41.028470 31221 raft_consensus.cc:504] T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:41.079124 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668b4a4d782c4278932dd8413dc3ab3e" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:41.079588 31221 leader_election.cc:288] T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:41.079756 29805 raft_consensus.cc:2454] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 3.
I20240501 20:01:41.066815 29728 raft_consensus.cc:2454] T c651044a305c4168805825a53d910899 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 2.
I20240501 20:01:41.080636 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b337b8e3e38d4ad6b1076fbd0d0a1609" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:41.081324 29805 raft_consensus.cc:2379] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 4: Already voted for candidate ee0d97247d394a8facfa774f15f3a67d in this term.
I20240501 20:01:40.999392 31220 raft_consensus.cc:482] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:41.082844 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e38a2e91a3d48d09a22abce723af833" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:41.086899 29807 raft_consensus.cc:2454] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 3.
I20240501 20:01:41.089094 31229 leader_election.cc:288] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:41.090663 29804 raft_consensus.cc:2454] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 5.
W20240501 20:01:41.094105 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 332475us
I20240501 20:01:41.043982 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:41.095352 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e38a2e91a3d48d09a22abce723af833" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:41.095604 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38cbeb0dc0c14e3d9182101eea885625" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:41.095757 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05df43bdfa479b902b1da297fb5974" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:41.097357 29734 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.505s user 0.021s sys 0.040s
I20240501 20:01:41.090772 31024 tablet_bootstrap.cc:492] T 77be7101c0fe4085b6706296f609870a P 888d99522e4844118f8f1cf26f268c8c: Bootstrap complete.
I20240501 20:01:41.099479 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d86f8848a184a87947fd927c44e5a35" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:41.101583 29804 raft_consensus.cc:2379] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 4: Already voted for candidate ee0d97247d394a8facfa774f15f3a67d in this term.
I20240501 20:01:41.101718 29805 raft_consensus.cc:2396] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240501 20:01:41.028992 30912 raft_consensus.cc:504] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:41.121961 29800 raft_consensus.cc:2396] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20240501 20:01:40.989300 31233 consensus_queue.cc:1035] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.124s
I20240501 20:01:41.102371 29700 leader_election.cc:302] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:41.109208 31229 consensus_queue.cc:1035] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.012s
I20240501 20:01:41.136334 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af08384057a4d85977481472f460168" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:41.146360 29807 raft_consensus.cc:2354] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 888d99522e4844118f8f1cf26f268c8c for earlier term 4. Current term is 5.
I20240501 20:01:41.165982 29656 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1ca3948261374cef8b1b206a43c00173"
dest_uuid: "4c8da1d580614449a636b855bf11b95f"
mode: GRACEFUL
new_leader_uuid: "fdfeac4190e045f290e003b2298d988d"
from {username='slave'} at 127.0.0.1:60736
I20240501 20:01:41.167650 31234 leader_election.cc:288] T c651044a305c4168805825a53d910899 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:41.168852 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 66589.3us
I20240501 20:01:41.170109 29700 leader_election.cc:302] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:41.117810 29653 raft_consensus.cc:2454] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 4.
W20240501 20:01:41.120105 29734 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.528s user 0.021s sys 0.040s
I20240501 20:01:41.091639 31220 raft_consensus.cc:3037] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Advancing to term 4
W20240501 20:01:41.121503 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 164532us
I20240501 20:01:41.123670 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb27193abbf4644a48662909bc7b04a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:41.119654 29729 raft_consensus.cc:2379] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 4: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:41.185838 29804 raft_consensus.cc:2454] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 3.
I20240501 20:01:41.112998 31209 leader_election.cc:288] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:41.125283 29805 raft_consensus.cc:1261] T a8787bc4d6b44649b9138d906730110a P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240501 20:01:41.115645 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac2171683884f6f97cdfeb0a656cb49" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:41.094008 31017 raft_consensus.cc:482] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:41.185880 29698 leader_election.cc:302] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:41.197016 31241 raft_consensus.cc:3037] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:41.197016 31234 raft_consensus.cc:2781] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:01:41.198166 29727 raft_consensus.cc:3037] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:41.242820 29801 raft_consensus.cc:1226] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [term 3 LEADER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 2. Current term is 3. Ops: []
I20240501 20:01:41.216696 31211 raft_consensus.cc:2781] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:41.197329 31042 leader_election.cc:288] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:41.280395 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c651044a305c4168805825a53d910899" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
W20240501 20:01:41.244220 31238 log.cc:924] Time spent T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c: Append to log took a long time: real 0.081s user 0.000s sys 0.001s
I20240501 20:01:41.283289 29625 leader_election.cc:302] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:41.286077 29700 leader_election.cc:302] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:41.288924 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5345f6eb62449ba69b4a9c2e349b31" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:41.289091 29700 leader_election.cc:302] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
W20240501 20:01:41.252624 29651 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37936 (request call id 79) took 2837 ms (client timeout 2074 ms). Trace:
W20240501 20:01:41.291046 29651 rpcz_store.cc:269] 0501 20:01:38.406053 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:38.475997 (+ 69944us) service_pool.cc:225] Handling call
0501 20:01:41.235993 (+2759996us) spinlock_profiling.cc:243] Waited 183 ms on lock 0x7b540032c3d4. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787f4fe 00007fe45c7bd9d7 00007fe455ea1c14 00007fe455ea1ba2 00007fe455ea1ad1 00007fe455ea1a4c 00007fe455ea0d02 00007fe455ca2bcc 00007fe455ca2006 00007fe455ca19af
0501 20:01:41.241528 (+ 5535us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":420334336}
I20240501 20:01:41.292659 30849 raft_consensus.cc:3037] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:41.297480 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5525f546426c47a38edf7f56ad04b7ab" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:41.298703 29653 raft_consensus.cc:2454] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 4.
I20240501 20:01:41.299516 29625 leader_election.cc:302] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:41.197940 31017 raft_consensus.cc:504] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:41.301332 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b337b8e3e38d4ad6b1076fbd0d0a1609" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:41.325609 29657 raft_consensus.cc:3037] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:41.318869 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e9f75387564275a60d38e9a910319c" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:41.319272 29772 leader_election.cc:302] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:41.253636 31211 raft_consensus.cc:482] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:41.334784 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458740ab5c444b6ab3d1991c69db5084" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:41.273211 31234 raft_consensus.cc:686] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d [term 3 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:41.361949 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5525fc4b82a44b0991c3dd6098fde81" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:41.367201 29805 raft_consensus.cc:3037] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:41.296270 31081 raft_consensus.cc:2781] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader election won for term 5
W20240501 20:01:41.370995 29772 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 51838.5us
I20240501 20:01:41.335039 31024 ts_tablet_manager.cc:1367] T 77be7101c0fe4085b6706296f609870a P 888d99522e4844118f8f1cf26f268c8c: Time spent bootstrapping tablet: real 1.910s user 0.028s sys 0.017s
I20240501 20:01:41.371906 29772 leader_election.cc:302] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:41.377247 29657 raft_consensus.cc:2454] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 4.
I20240501 20:01:41.377571 29772 leader_election.cc:302] T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:41.375913 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca3948261374cef8b1b206a43c00173" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:41.378821 29800 raft_consensus.cc:2454] T 1ca3948261374cef8b1b206a43c00173 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 1.
I20240501 20:01:41.382696 31165 leader_election.cc:288] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:41.329476 31063 raft_consensus.cc:2781] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240501 20:01:41.384295 31063 raft_consensus.cc:482] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:41.384716 31063 raft_consensus.cc:3037] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:41.328563 31209 raft_consensus.cc:482] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:41.333894 31151 raft_consensus.cc:2726] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:41.350104 31211 raft_consensus.cc:3037] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:41.329434 31053 raft_consensus.cc:2781] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240501 20:01:41.404183 30815 raft_consensus.cc:2781] T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader election won for term 4
W20240501 20:01:41.405604 29699 service_pool.cc:135] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items. [suppressed 1 similar messages]
I20240501 20:01:41.407845 29772 leader_election.cc:302] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
W20240501 20:01:41.415344 29625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 114715us
I20240501 20:01:41.357426 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f261d2a05bf4c838532049601694b4a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:41.420070 29625 leader_election.cc:302] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:41.424597 29625 leader_election.cc:302] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:41.424772 29772 leader_election.cc:302] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:41.426134 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211e27a3b0084b89a6dd7550849b5a65" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:41.426954 29800 raft_consensus.cc:2454] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 2.
I20240501 20:01:41.428035 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb27193abbf4644a48662909bc7b04a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:41.388607 31202 raft_consensus.cc:2781] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:41.430418 29800 raft_consensus.cc:2454] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 3.
I20240501 20:01:41.445128 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcf30cf66714efdac997862dafbe4a2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:41.446718 29807 raft_consensus.cc:2396] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240501 20:01:41.399932 31118 raft_consensus.cc:2781] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:41.454632 29655 raft_consensus.cc:1261] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Refusing update from remote peer 888d99522e4844118f8f1cf26f268c8c: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240501 20:01:41.456744 29700 leader_election.cc:302] T 1ca3948261374cef8b1b206a43c00173 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
W20240501 20:01:41.414212 31143 consensus_meta.cc:287] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d: Time spent flushing consensus metadata: real 0.538s user 0.010s sys 0.000s
I20240501 20:01:41.464773 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36057b3f24d44b9aac3a1378053a395f" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:41.467346 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49da4246f5f6438abe0e7e8c9a523d80" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:41.467895 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e49dd5180f04313ab33942fb3ddc147" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:41.469748 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e38a2e91a3d48d09a22abce723af833" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:41.473049 29657 raft_consensus.cc:2454] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 3.
I20240501 20:01:41.372362 31233 consensus_queue.cc:1046] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.093s
I20240501 20:01:41.471932 29700 leader_election.cc:302] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:41.470188 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458740ab5c444b6ab3d1991c69db5084" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:41.431007 31053 raft_consensus.cc:482] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:41.457557 31165 raft_consensus.cc:2781] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:01:41.475014 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:41.477634 29657 raft_consensus.cc:1261] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240501 20:01:41.477569 29700 leader_election.cc:302] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:41.452271 31202 raft_consensus.cc:482] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:41.476001 29807 raft_consensus.cc:2454] T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 3.
I20240501 20:01:41.452706 29651 raft_consensus.cc:1261] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240501 20:01:41.457790 31194 raft_consensus.cc:2781] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240501 20:01:41.408541 31209 raft_consensus.cc:504] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:41.458387 30815 raft_consensus.cc:686] T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d [term 4 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:41.458839 31118 raft_consensus.cc:482] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:41.480365 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c651044a305c4168805825a53d910899" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:41.465401 29729 raft_consensus.cc:2454] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 3.
I20240501 20:01:41.498086 31165 raft_consensus.cc:686] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [term 4 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:41.500836 29654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d86f8848a184a87947fd927c44e5a35" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:41.501533 29654 raft_consensus.cc:2454] T 2d86f8848a184a87947fd927c44e5a35 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 3.
I20240501 20:01:41.501816 31017 leader_election.cc:288] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:41.498380 30753 raft_consensus.cc:2781] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:41.494534 31194 raft_consensus.cc:482] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:41.502835 31118 raft_consensus.cc:3037] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:41.484822 31144 consensus_queue.cc:1035] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:01:41.485314 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d2cac25f354a97b1bfb53d57ae7c08" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:41.457645 29727 raft_consensus.cc:2454] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 4.
I20240501 20:01:41.486246 31250 raft_consensus.cc:2781] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:41.502727 30753 raft_consensus.cc:482] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240501 20:01:41.504601 30384 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 343251us
I20240501 20:01:41.482542 29772 leader_election.cc:302] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:41.489105 31205 raft_consensus.cc:2781] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:41.489681 29806 raft_consensus.cc:2454] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 4.
W20240501 20:01:41.509703 31247 log.cc:924] Time spent T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.078s user 0.000s sys 0.001s
I20240501 20:01:41.509905 31205 raft_consensus.cc:482] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:41.509981 31020 raft_consensus.cc:482] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:41.505764 31250 raft_consensus.cc:482] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:41.510413 31205 raft_consensus.cc:3037] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:41.510694 31250 raft_consensus.cc:3037] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:41.477854 31053 raft_consensus.cc:3037] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:41.480753 31202 raft_consensus.cc:3037] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:41.500917 31209 leader_election.cc:288] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:41.500568 31165 consensus_queue.cc:237] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:41.502890 31194 raft_consensus.cc:3037] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:41.475908 31245 raft_consensus.cc:2781] T 1ca3948261374cef8b1b206a43c00173 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:01:41.504987 29807 raft_consensus.cc:3037] T d1d2cac25f354a97b1bfb53d57ae7c08 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:41.493517 31085 raft_consensus.cc:2781] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election won for term 4
W20240501 20:01:41.507138 31217 log.cc:924] Time spent T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d: Append to log took a long time: real 1.032s user 0.003s sys 0.000s
I20240501 20:01:41.494789 29700 leader_election.cc:302] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:41.453923 29699 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 799)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 809)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 813)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 814)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 811)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 817)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 887)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 801)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 790)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 812)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 920)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 883)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 884)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 886)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 890)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 892)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 791)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 898)
I20240501 20:01:41.509696 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb27193abbf4644a48662909bc7b04a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:41.518008 29772 leader_election.cc:302] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:41.510541 31020 raft_consensus.cc:504] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:41.590057 31209 raft_consensus.cc:2781] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election won for term 4
W20240501 20:01:41.523821 29625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 99334us
I20240501 20:01:41.472201 31143 raft_consensus.cc:504] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:41.507197 29805 raft_consensus.cc:2454] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 4.
I20240501 20:01:41.506246 30815 consensus_queue.cc:237] T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:41.590507 31209 raft_consensus.cc:482] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:41.508580 31063 raft_consensus.cc:504] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:41.542594 31221 consensus_queue.cc:1035] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.009s
I20240501 20:01:41.551556 31245 raft_consensus.cc:482] T 1ca3948261374cef8b1b206a43c00173 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
W20240501 20:01:41.553534 31223 consensus_meta.cc:287] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c: Time spent flushing consensus metadata: real 0.589s user 0.010s sys 0.000s
I20240501 20:01:41.540297 31017 consensus_queue.cc:1035] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.014s
I20240501 20:01:41.554663 31085 raft_consensus.cc:482] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:41.506764 30753 raft_consensus.cc:3037] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Advancing to term 4
W20240501 20:01:41.555846 29802 consensus_meta.cc:287] T 456035a1cb3f4f7da52d0fc1dd1c656c P ee0d97247d394a8facfa774f15f3a67d: Time spent flushing consensus metadata: real 0.529s user 0.007s sys 0.003s
I20240501 20:01:41.505611 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48630ae6eb394ac1b54955dd01cf0ed2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
W20240501 20:01:41.591136 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 96476us
W20240501 20:01:41.593441 29772 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 75545.1us
I20240501 20:01:41.595724 29804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41858 (request call id 950) took 1014 ms. Trace:
I20240501 20:01:41.508497 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77be7101c0fe4085b6706296f609870a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:41.521409 31019 raft_consensus.cc:2781] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:41.611814 31019 raft_consensus.cc:482] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:41.611950 29729 raft_consensus.cc:2379] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 3: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:41.612259 31019 raft_consensus.cc:3037] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:41.612924 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9046e38fc57f4364aa3d539e1fc01928" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:41.613461 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c597f15e844f30837012baed4b0501" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:41.613796 29729 raft_consensus.cc:2379] T 9046e38fc57f4364aa3d539e1fc01928 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 3: Already voted for candidate 888d99522e4844118f8f1cf26f268c8c in this term.
I20240501 20:01:41.516422 31081 raft_consensus.cc:686] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [term 5 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:41.477380 31234 consensus_queue.cc:237] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:41.596958 29625 leader_election.cc:302] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:41.599035 29800 raft_consensus.cc:1226] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [term 4 LEADER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 3. Current term is 4. Ops: []
I20240501 20:01:41.600217 29803 raft_consensus.cc:2379] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 5: Already voted for candidate ee0d97247d394a8facfa774f15f3a67d in this term.
I20240501 20:01:41.599946 31205 raft_consensus.cc:504] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:41.607537 29700 leader_election.cc:302] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:41.608417 29651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfac893b4b4f457694a4eb184b6abb53" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:41.610848 31024 raft_consensus.cc:348] T 77be7101c0fe4085b6706296f609870a P 888d99522e4844118f8f1cf26f268c8c [term 2 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:41.615751 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:41.622519 29657 raft_consensus.cc:2454] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 4.
W20240501 20:01:41.616964 29772 consensus_peers.cc:485] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d -> Peer 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227): Couldn't send request to peer 888d99522e4844118f8f1cf26f268c8c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:41.605311 31085 raft_consensus.cc:3037] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:41.614625 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c651044a305c4168805825a53d910899" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:41.602248 31223 raft_consensus.cc:504] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:41.607717 29804 rpcz_store.cc:276] 0501 20:01:40.581284 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:41.190138 (+608854us) service_pool.cc:225] Handling call
0501 20:01:41.190449 (+ 311us) raft_consensus.cc:1457] Updating replica for 1 ops
0501 20:01:41.190672 (+ 223us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:41.190704 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0501 20:01:41.222611 (+ 31907us) log.cc:842] Serialized 26 byte log entry
0501 20:01:41.588798 (+366187us) raft_consensus.cc:1631] Marking committed up to 1
0501 20:01:41.588834 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:41.588910 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0501 20:01:41.595449 (+ 6539us) raft_consensus.cc:1671] finished
0501 20:01:41.595470 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:41.595600 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4691,"thread_start_us":365758,"threads_started":1,"wal-append.queue_time_us":366073}
I20240501 20:01:41.599471 31245 raft_consensus.cc:3037] T 1ca3948261374cef8b1b206a43c00173 P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:41.596818 31209 raft_consensus.cc:3037] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:41.697173 31053 raft_consensus.cc:504] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:41.698426 31241 raft_consensus.cc:2726] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:41.698793 30912 leader_election.cc:288] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
W20240501 20:01:41.683765 29625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 86607.8us
I20240501 20:01:41.671262 31205 leader_election.cc:288] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:41.703629 31255 log.cc:924] Time spent T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.104s user 0.001s sys 0.001s
I20240501 20:01:41.689528 29808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41858 (request call id 927) took 2174 ms. Trace:
I20240501 20:01:41.641937 29807 raft_consensus.cc:2454] T d1d2cac25f354a97b1bfb53d57ae7c08 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 4.
I20240501 20:01:41.624873 29772 leader_election.cc:302] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:41.630277 29700 leader_election.cc:302] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:41.704151 29808 rpcz_store.cc:276] 0501 20:01:39.514670 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:39.516306 (+ 1636us) service_pool.cc:225] Handling call
0501 20:01:41.688723 (+2172417us) spinlock_profiling.cc:243] Waited 14.2 ms on lock 0x7b58000d2c98. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe454797b51 00007fe4549193b1 00007fe454919262 00007fe45d80b889 00007fe45c7483c0 00007fe45c80e3bb 00007fe45c80e441 00007fe45c7d2060 00007fe45c7bd5a9 00007fe45c7bd052 00007fe455e9f694
0501 20:01:41.688937 (+ 214us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:41.689104 (+ 167us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:41.689144 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:41.689207 (+ 63us) raft_consensus.cc:1631] Marking committed up to 1
0501 20:01:41.689250 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:41.689327 (+ 77us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:41.689425 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":32711424}
W20240501 20:01:41.705788 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 75616.2us
W20240501 20:01:41.706187 29772 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 81378.2us
I20240501 20:01:41.596119 31246 raft_consensus.cc:2726] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240501 20:01:41.710512 29700 leader_election.cc:302] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:41.606119 29802 raft_consensus.cc:2454] T 456035a1cb3f4f7da52d0fc1dd1c656c P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 3.
I20240501 20:01:41.669574 31081 consensus_queue.cc:237] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:41.712316 29804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41858 (request call id 955) took 1038 ms. Trace:
I20240501 20:01:41.710566 29807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41858 (request call id 954) took 1058 ms. Trace:
I20240501 20:01:41.713277 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c597f15e844f30837012baed4b0501" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:41.642935 31249 raft_consensus.cc:2781] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:41.699391 31143 leader_election.cc:288] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:41.716966 30414 raft_consensus.cc:1226] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [term 4 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 3. Current term is 4. Ops: []
I20240501 20:01:41.715353 31253 raft_consensus.cc:2781] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:01:41.708971 31017 raft_consensus.cc:2781] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240501 20:01:41.700466 31063 leader_election.cc:288] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:41.642040 31232 raft_consensus.cc:2781] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:41.716236 29807 rpcz_store.cc:276] 0501 20:01:40.652376 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:41.709901 (+1057525us) service_pool.cc:225] Handling call
0501 20:01:41.710137 (+ 236us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:41.710292 (+ 155us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:41.710315 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:41.710351 (+ 36us) raft_consensus.cc:1631] Marking committed up to 2
0501 20:01:41.710378 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:41.710424 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:41.710510 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:41.699347 30912 raft_consensus.cc:482] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:41.701774 31053 leader_election.cc:288] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:41.715929 29804 rpcz_store.cc:276] 0501 20:01:40.673738 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:41.709380 (+1035642us) service_pool.cc:225] Handling call
0501 20:01:41.711864 (+ 2484us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:41.712017 (+ 153us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:41.712072 (+ 55us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:41.712109 (+ 37us) raft_consensus.cc:1631] Marking committed up to 2
0501 20:01:41.712135 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:41.712185 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:41.712270 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:41.685079 31024 raft_consensus.cc:729] T 77be7101c0fe4085b6706296f609870a P 888d99522e4844118f8f1cf26f268c8c [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 888d99522e4844118f8f1cf26f268c8c, State: Initialized, Role: LEARNER
I20240501 20:01:41.623853 29651 raft_consensus.cc:2454] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 4.
I20240501 20:01:41.709753 31205 raft_consensus.cc:2781] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:01:41.734902 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211e27a3b0084b89a6dd7550849b5a65" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:41.757285 29808 raft_consensus.cc:1226] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [term 4 LEADER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 3. Current term is 4. Ops: []
I20240501 20:01:41.666452 31220 raft_consensus.cc:504] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:41.750042 29804 raft_consensus.cc:1226] T 456035a1cb3f4f7da52d0fc1dd1c656c P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 2. Current term is 3. Ops: []
I20240501 20:01:41.720894 31249 raft_consensus.cc:482] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:41.725059 29727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 887) took 1095 ms. Trace:
I20240501 20:01:41.726641 31017 raft_consensus.cc:482] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:41.727855 31232 raft_consensus.cc:482] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:41.720310 31202 raft_consensus.cc:504] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:41.778755 29772 leader_election.cc:302] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:41.720455 30753 raft_consensus.cc:504] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:41.786468 29625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 59214.3us
I20240501 20:01:41.744160 31253 raft_consensus.cc:686] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:41.788376 29732 raft_consensus.cc:2379] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 4: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:41.777988 31249 raft_consensus.cc:3037] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:41.780541 31017 raft_consensus.cc:3037] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:41.777897 31205 raft_consensus.cc:686] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:41.749159 30912 raft_consensus.cc:504] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:41.783823 31232 raft_consensus.cc:3037] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:41.799650 29735 raft_consensus.cc:2354] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d for earlier term 3. Current term is 4.
I20240501 20:01:41.801185 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcf30cf66714efdac997862dafbe4a2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:41.801924 31202 leader_election.cc:288] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:41.800215 29699 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 801)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 790)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 812)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 920)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 883)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 884)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 886)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 890)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 892)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 791)
I20240501 20:01:41.788200 30815 raft_consensus.cc:2781] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240501 20:01:41.777696 29727 rpcz_store.cc:276] 0501 20:01:40.625943 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:41.614367 (+988424us) service_pool.cc:225] Handling call
0501 20:01:41.720974 (+106607us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:41.806438 29625 leader_election.cc:302] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:41.812783 30815 raft_consensus.cc:482] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:41.817353 31256 consensus_queue.cc:1046] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.040s
I20240501 20:01:41.817958 30815 raft_consensus.cc:3037] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:41.818128 31256 consensus_queue.cc:1225] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer ee0d97247d394a8facfa774f15f3a67d log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20240501 20:01:41.807598 30849 raft_consensus.cc:482] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:41.820369 30849 raft_consensus.cc:504] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:41.821813 29658 raft_consensus.cc:2354] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d for earlier term 4. Current term is 5.
I20240501 20:01:41.823045 29772 leader_election.cc:398] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 election: Vote denied by peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959) with higher term. Message: Invalid argument: T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d for earlier term 4. Current term is 5.
I20240501 20:01:41.823278 29772 leader_election.cc:401] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20240501 20:01:41.834105 31267 raft_consensus.cc:2781] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:01:41.834769 31267 raft_consensus.cc:686] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:41.835361 31267 consensus_queue.cc:237] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:41.789578 31024 consensus_queue.cc:260] T 77be7101c0fe4085b6706296f609870a P 888d99522e4844118f8f1cf26f268c8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:41.844533 29699 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 791) took 1147 ms. Trace:
I20240501 20:01:41.793327 31252 consensus_queue.cc:1046] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.037s
I20240501 20:01:41.844834 29699 rpcz_store.cc:276] 0501 20:01:40.697526 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:41.844496 (+1146970us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240501 20:01:41.850545 31243 raft_consensus.cc:3037] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:41.850857 30849 leader_election.cc:288] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:41.832990 31223 leader_election.cc:288] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 4 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:41.852648 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649b71f9f267437ca51c758dd9ca6987" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:41.851938 29699 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36260 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 790)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 812)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 920)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 883)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 884)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 886)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 890)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 892)
I20240501 20:01:41.853330 29655 raft_consensus.cc:2454] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 4.
I20240501 20:01:41.857961 29772 leader_election.cc:302] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:41.830801 31253 consensus_queue.cc:237] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:41.846709 31262 consensus_queue.cc:1035] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.139s
I20240501 20:01:41.861160 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0374bb790ed345ed90452595f71383a2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:41.861725 29655 raft_consensus.cc:2454] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 3.
I20240501 20:01:41.849258 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca3948261374cef8b1b206a43c00173" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:41.864336 29699 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 892) took 1237 ms. Trace:
I20240501 20:01:41.863696 29772 leader_election.cc:302] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:41.866346 29699 rpcz_store.cc:276] 0501 20:01:40.626798 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:41.864267 (+1237469us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240501 20:01:41.866876 29699 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 790)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 812)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 920)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 883)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 884)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 886)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 890)
I20240501 20:01:41.871202 30753 leader_election.cc:288] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:41.871870 30912 leader_election.cc:288] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:41.853319 31216 raft_consensus.cc:482] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:41.880993 29699 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 890) took 1254 ms. Trace:
I20240501 20:01:41.881667 29699 rpcz_store.cc:276] 0501 20:01:40.626470 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:41.880953 (+1254483us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240501 20:01:41.861585 31221 raft_consensus.cc:2781] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240501 20:01:41.882329 29699 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 790)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 812)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 920)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 883)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 884)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 886)
I20240501 20:01:41.843593 31205 consensus_queue.cc:237] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:41.863085 31220 leader_election.cc:288] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:41.881718 30849 raft_consensus.cc:2781] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:41.890179 31024 ts_tablet_manager.cc:1398] T 77be7101c0fe4085b6706296f609870a P 888d99522e4844118f8f1cf26f268c8c: Time spent starting tablet: real 0.488s user 0.006s sys 0.010s
I20240501 20:01:41.890961 31019 raft_consensus.cc:504] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:41.893395 29699 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 886) took 1267 ms. Trace:
W20240501 20:01:41.895150 31269 log.cc:924] Time spent T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20240501 20:01:41.900265 29699 rpcz_store.cc:276] 0501 20:01:40.625763 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:41.893350 (+1267587us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240501 20:01:41.903679 31019 leader_election.cc:288] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:41.903520 30815 raft_consensus.cc:504] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:41.897181 30849 raft_consensus.cc:482] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
W20240501 20:01:41.900847 31024 ts_tablet_manager.cc:1430] T 77be7101c0fe4085b6706296f609870a P 888d99522e4844118f8f1cf26f268c8c: Tablet startup took 521ms
I20240501 20:01:41.911854 29699 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 790)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 812)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 57)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 920)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 883)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 884)
I20240501 20:01:41.920712 29699 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 884) took 1298 ms. Trace:
I20240501 20:01:41.921447 29699 rpcz_store.cc:276] 0501 20:01:40.622300 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:41.920664 (+1298364us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240501 20:01:41.884284 31250 raft_consensus.cc:504] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:41.913764 31085 raft_consensus.cc:504] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:41.925958 30935 raft_consensus.cc:3037] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:41.926491 31250 leader_election.cc:288] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:41.914378 30849 raft_consensus.cc:3037] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:41.929419 31181 raft_consensus.cc:3032] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:01:41.886139 31221 raft_consensus.cc:482] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:41.934749 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb27193abbf4644a48662909bc7b04a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:41.937280 29804 raft_consensus.cc:3037] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:41.922372 29699 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 790)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 812)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 57)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 920)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 883)
I20240501 20:01:41.900427 31262 consensus_queue.cc:1046] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20240501 20:01:41.897471 31216 raft_consensus.cc:504] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:41.940866 31221 raft_consensus.cc:3037] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:41.942375 29699 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 883) took 1322 ms. Trace:
I20240501 20:01:41.926827 31085 leader_election.cc:288] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:41.947290 29699 rpcz_store.cc:276] 0501 20:01:40.619755 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:41.942330 (+1322575us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240501 20:01:41.948897 29699 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 790)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 812)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 57)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 920)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 59)
I20240501 20:01:41.936110 31181 raft_consensus.cc:729] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:41.943825 29654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e38a2e91a3d48d09a22abce723af833" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:41.963459 29699 service_pool.cc:138] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:47414 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 790)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 812)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 57)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 920)
I20240501 20:01:41.948433 30815 leader_election.cc:288] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:41.909678 31209 raft_consensus.cc:504] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:41.972455 31085 raft_consensus.cc:482] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:41.973109 31216 leader_election.cc:288] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:41.973575 31262 raft_consensus.cc:482] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:41.972919 29699 service_pool.cc:138] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 790)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 812)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 57)
I20240501 20:01:41.980333 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a35893b91247a99912eba787d787cf" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:41.988826 29660 raft_consensus.cc:2454] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 3.
I20240501 20:01:41.992200 29772 leader_election.cc:302] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:41.988332 29699 service_pool.cc:138] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 790)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 812)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 57)
I20240501 20:01:41.976583 31214 leader_election.cc:288] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:41.995469 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38cbeb0dc0c14e3d9182101eea885625" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:41.997802 29658 raft_consensus.cc:3037] T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:41.994340 29699 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 790)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 812)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 57)
W20240501 20:01:42.000571 29624 leader_election.cc:334] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 1.722s (SENT)
I20240501 20:01:42.001974 29624 leader_election.cc:302] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:42.000275 29699 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 790)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 812)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 57)
I20240501 20:01:42.008298 29699 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 790)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 812)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 57)
I20240501 20:01:42.001353 30815 raft_consensus.cc:2781] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:41.971074 31017 raft_consensus.cc:504] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:42.023511 29699 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 790)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 812)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 57)
I20240501 20:01:42.005648 29654 raft_consensus.cc:2379] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 4: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:42.006764 31250 raft_consensus.cc:2726] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240501 20:01:42.008666 31245 raft_consensus.cc:504] T 1ca3948261374cef8b1b206a43c00173 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:42.037986 29624 consensus_peers.cc:485] T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f -> Peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Couldn't send request to peer fdfeac4190e045f290e003b2298d988d. Status: Remote error: Service unavailable: UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items.. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:41.982184 31118 raft_consensus.cc:504] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:41.987807 31085 raft_consensus.cc:504] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:42.038550 29624 consensus_peers.cc:485] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f -> Peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Couldn't send request to peer fdfeac4190e045f290e003b2298d988d. Status: Remote error: Service unavailable: UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items.. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:42.035071 29699 service_pool.cc:138] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:47414 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 790)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 812)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 939)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60)
I20240501 20:01:42.007431 31181 consensus_queue.cc:260] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:42.048781 29808 raft_consensus.cc:1226] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 4. Current term is 5. Ops: []
W20240501 20:01:42.040428 29624 consensus_peers.cc:485] T 1ca3948261374cef8b1b206a43c00173 P 4c8da1d580614449a636b855bf11b95f -> Peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Couldn't send request to peer fdfeac4190e045f290e003b2298d988d. Status: Remote error: Service unavailable: UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items.. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:41.994364 31262 raft_consensus.cc:504] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:42.051394 29624 consensus_peers.cc:485] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f -> Peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Couldn't send request to peer fdfeac4190e045f290e003b2298d988d. Status: Remote error: Service unavailable: UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items.. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:42.029474 30815 raft_consensus.cc:482] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:42.055774 30815 raft_consensus.cc:3037] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Advancing to term 4
W20240501 20:01:42.074064 29806 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41858 (request call id 931) took 2531 ms (client timeout 1801 ms). Trace:
I20240501 20:01:42.065613 29727 raft_consensus.cc:2379] T 1ca3948261374cef8b1b206a43c00173 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 2: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
W20240501 20:01:42.074615 29806 rpcz_store.cc:269] 0501 20:01:39.542278 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:39.881226 (+338948us) service_pool.cc:225] Handling call
0501 20:01:42.073839 (+2192613us) spinlock_profiling.cc:243] Waited 1.61 s on lock 0x7b5400375fd4. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787f4fe 00007fe45c7bd9d7 00007fe455ea1c14 00007fe455ea1ba2 00007fe455ea1ad1 00007fe455ea1a4c 00007fe455ea0d02 00007fe455ca2bcc 00007fe455ca2006 00007fe455ca19af
0501 20:01:42.073963 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3696188288}
I20240501 20:01:42.077489 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3f29f0eeea4b119984634202e5d9cd" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:42.078742 29806 raft_consensus.cc:1226] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 3. Current term is 4. Ops: []
I20240501 20:01:42.083155 29658 raft_consensus.cc:2454] T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 4.
I20240501 20:01:42.080432 29699 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 790)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 812)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 939)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60)
W20240501 20:01:42.086112 29624 consensus_peers.cc:485] T 3c5345f6eb62449ba69b4a9c2e349b31 P 4c8da1d580614449a636b855bf11b95f -> Peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Couldn't send request to peer fdfeac4190e045f290e003b2298d988d. Status: Remote error: Service unavailable: UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items.. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:42.079942 29806 raft_consensus.cc:1226] T 3c5345f6eb62449ba69b4a9c2e349b31 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 2. Current term is 3. Ops: []
I20240501 20:01:42.093649 29699 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 790)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 812)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 939)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60)
I20240501 20:01:42.102694 29806 raft_consensus.cc:1226] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20240501 20:01:42.113468 29804 raft_consensus.cc:2454] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 4.
I20240501 20:01:42.114866 29700 leader_election.cc:302] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:42.098831 29699 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 790)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 812)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 939)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60)
I20240501 20:01:42.115775 31079 raft_consensus.cc:2781] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:01:42.116243 31079 raft_consensus.cc:686] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:42.117029 31079 consensus_queue.cc:237] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:42.118618 29624 consensus_peers.cc:485] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f -> Peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Couldn't send request to peer fdfeac4190e045f290e003b2298d988d. Status: Remote error: Service unavailable: UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items.. This is attempt 1: this message will repeat every 5th retry.
W20240501 20:01:42.120414 29624 leader_election.cc:334] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 1.865s (SENT)
I20240501 20:01:42.120898 29624 leader_election.cc:302] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
W20240501 20:01:42.122066 29606 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica ee0d97247d394a8facfa774f15f3a67d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } }
I20240501 20:01:42.118937 29699 service_pool.cc:138] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 790)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 812)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 939)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60)
W20240501 20:01:41.920444 31024 ts_tablet_manager.cc:1432] T 77be7101c0fe4085b6706296f609870a P 888d99522e4844118f8f1cf26f268c8c: Trace:
0501 20:01:35.505828 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:35.508894 (+ 3066us) service_pool.cc:225] Handling call
0501 20:01:35.713300 (+204406us) ts_tablet_manager.cc:903] T 77be7101c0fe4085b6706296f609870a P 888d99522e4844118f8f1cf26f268c8c: Initiating tablet copy from peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
0501 20:01:36.731493 (+1018193us) tablet_copy_client.cc:353] Tablet copy session begun
0501 20:01:37.062776 (+331283us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:37.063053 (+ 277us) tablet_copy_client.cc:482] Wrote new tablet metadata
0501 20:01:37.412320 (+349267us) tablet_copy_client.cc:403] Wrote new consensus metadata
0501 20:01:37.412742 (+ 422us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:37.413099 (+ 357us) ts_tablet_manager.cc:958] Replica registered.
0501 20:01:37.413167 (+ 68us) inbound_call.cc:173] Queueing success response
0501 20:01:37.619016 (+205849us) spinlock_profiling.cc:243] Waited 1.31 ms on lock 0x7b18002fcc80. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe45d7f2661 00007fe45d7ff3e1 00007fe4549511b8 00007fe454954691 00007fe454954649 00007fe4549545d9 00007fe4549545a1 00007fe45495389d 00007fe45dbc5da4 00007fe45dbc5bd9 00007fe454933936
0501 20:01:39.425019 (+1806003us) ts_tablet_manager.cc:1343] Bootstrapping tablet
0501 20:01:41.408358 (+1983339us) ts_tablet_manager.cc:1399] Starting tablet replica
0501 20:01:41.608736 (+200378us) tablet_replica.cc:259] Starting instrumentation
0501 20:01:41.609361 (+ 625us) tablet_replica.cc:278] Starting consensus
Metrics: {"dns_us":272,"mutex_wait_us":2234,"raft.queue_time_us":2642,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":7350,"spinlock_wait_cycles":3022080,"tablet-copy.queue_time_us":140782,"tablet-download-pool-77be7101c0fe4085b6706296f609870a.queue_time_us":34419,"tablet-download-pool-77be7101c0fe4085b6706296f609870a.run_cpu_time_us":1436,"tablet-download-pool-77be7101c0fe4085b6706296f609870a.run_wall_time_us":1537423,"thread_start_us":744961,"threads_started":5}
W20240501 20:01:42.129576 29606 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 8edcd903224d48f889b46e1b5318e0ec, TS fdfeac4190e045f290e003b2298d988d -> TS 888d99522e4844118f8f1cf26f268c8c move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:42.129979 29606 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 8edcd903224d48f889b46e1b5318e0ec, TS fdfeac4190e045f290e003b2298d988d -> TS 888d99522e4844118f8f1cf26f268c8c move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:42.091377 30383 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 103423us
W20240501 20:01:42.133919 31211 consensus_meta.cc:287] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d: Time spent flushing consensus metadata: real 0.698s user 0.001s sys 0.017s
I20240501 20:01:42.089658 31181 raft_consensus.cc:3037] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:42.090132 31152 leader_election.cc:288] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:42.142016 31132 ts_tablet_manager.cc:1875] T be383be783234855878cb8fd442c7b05 P 888d99522e4844118f8f1cf26f268c8c: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 20:01:42.142450 31017 leader_election.cc:288] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:42.142740 31118 leader_election.cc:288] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:42.139895 31085 leader_election.cc:288] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:42.140331 31262 leader_election.cc:288] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:42.140424 31194 consensus_meta.cc:287] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d: Time spent flushing consensus metadata: real 0.584s user 0.013s sys 0.008s
I20240501 20:01:42.136206 31245 leader_election.cc:288] T 1ca3948261374cef8b1b206a43c00173 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:42.135139 30650 raft_consensus.cc:482] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:42.145743 30935 raft_consensus.cc:2726] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240501 20:01:42.135843 31209 leader_election.cc:288] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:42.134328 31211 raft_consensus.cc:504] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:42.135188 30753 consensus_queue.cc:1046] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.014s
I20240501 20:01:42.152004 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca3948261374cef8b1b206a43c00173" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:42.153040 29804 raft_consensus.cc:3037] T 1ca3948261374cef8b1b206a43c00173 P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:42.125932 29699 service_pool.cc:138] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 790)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 812)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 939)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60)
W20240501 20:01:42.149358 29606 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 77be7101c0fe4085b6706296f609870a, TS 4c8da1d580614449a636b855bf11b95f -> TS 888d99522e4844118f8f1cf26f268c8c move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:42.155634 29606 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 77be7101c0fe4085b6706296f609870a, TS 4c8da1d580614449a636b855bf11b95f -> TS 888d99522e4844118f8f1cf26f268c8c move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:42.149475 30906 raft_consensus.cc:482] T 010ceabf3e5a4b1c9330b80dade765bb P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:42.155316 29699 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 790)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 812)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 939)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60)
I20240501 20:01:42.153671 30650 raft_consensus.cc:504] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:42.160377 29699 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 790)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 812)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 939)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60)
I20240501 20:01:42.163261 29652 raft_consensus.cc:2454] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 3.
W20240501 20:01:42.165936 29624 consensus_peers.cc:485] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f -> Peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Couldn't send request to peer fdfeac4190e045f290e003b2298d988d. Status: Remote error: Service unavailable: UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items.. This is attempt 1: this message will repeat every 5th retry.
W20240501 20:01:42.167554 31276 log.cc:924] Time spent T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.095s user 0.000s sys 0.001s
I20240501 20:01:42.169593 31178 leader_election.cc:288] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
W20240501 20:01:42.173184 31280 log.cc:924] Time spent T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.163s user 0.000s sys 0.001s
W20240501 20:01:42.176309 29624 leader_election.cc:334] T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 1.925s (SENT)
I20240501 20:01:42.176714 29624 leader_election.cc:302] T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:42.179827 29730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 882) took 1562 ms. Trace:
I20240501 20:01:42.161267 30906 raft_consensus.cc:504] T 010ceabf3e5a4b1c9330b80dade765bb P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:42.180222 29730 rpcz_store.cc:276] 0501 20:01:40.617342 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:40.617475 (+ 133us) service_pool.cc:225] Handling call
0501 20:01:40.625309 (+ 7834us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:40.827971 (+202662us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:42.179622 (+1351651us) spinlock_profiling.cc:243] Waited 188 ms on lock 0x7b5400142654. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787c1a1 00007fe45787abee 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc 00007fe455e9e782 00007fe455ca2bcc 00007fe455ca2006
0501 20:01:42.179780 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":434541824}
I20240501 20:01:42.181871 30906 leader_election.cc:288] T 010ceabf3e5a4b1c9330b80dade765bb P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:42.154413 30815 raft_consensus.cc:504] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:42.183226 31243 raft_consensus.cc:2726] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959) with higher term. Message: Invalid argument: T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d for earlier term 4. Current term is 5.
I20240501 20:01:42.184013 30815 leader_election.cc:288] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:42.152889 31194 raft_consensus.cc:504] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:42.185300 29806 raft_consensus.cc:1226] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 3. Current term is 4. Ops: []
W20240501 20:01:42.185365 29772 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.27.217.196:37227, user_credentials={real_user=slave}} blocked reactor thread for 190175us
I20240501 20:01:42.186771 31232 raft_consensus.cc:504] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:42.196256 29606 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet d48fd0a8e289401aa7417c42ff2c81b2, TS fdfeac4190e045f290e003b2298d988d -> TS 888d99522e4844118f8f1cf26f268c8c move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:42.196645 29606 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet d48fd0a8e289401aa7417c42ff2c81b2, TS fdfeac4190e045f290e003b2298d988d -> TS 888d99522e4844118f8f1cf26f268c8c move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:42.197876 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d86f8848a184a87947fd927c44e5a35" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:42.198388 29657 raft_consensus.cc:3037] T 2d86f8848a184a87947fd927c44e5a35 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:42.166335 29699 service_pool.cc:138] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 790)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 812)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 939)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60)
I20240501 20:01:42.199493 29658 raft_consensus.cc:1261] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240501 20:01:42.200234 29699 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 790)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 812)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 939)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60)
I20240501 20:01:42.177604 31250 raft_consensus.cc:2726] T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240501 20:01:42.208456 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:42.211458 29659 raft_consensus.cc:3037] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:42.213644 31211 leader_election.cc:288] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:42.215894 31194 leader_election.cc:288] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:42.218238 31106 leader_election.cc:288] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 5 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:42.221165 30849 raft_consensus.cc:504] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:42.221655 30414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40670 (request call id 50) took 1462 ms. Trace:
I20240501 20:01:42.223835 29772 leader_election.cc:302] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:42.203541 29699 service_pool.cc:138] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 790)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 812)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 939)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60)
I20240501 20:01:42.222055 30414 rpcz_store.cc:276] 0501 20:01:40.770781 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:40.780302 (+ 9521us) service_pool.cc:225] Handling call
0501 20:01:40.813563 (+ 33261us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:41.728618 (+915055us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:42.221429 (+492811us) spinlock_profiling.cc:243] Waited 896 ms on lock 0x7b54005ce754. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787c1a1 00007fe45787abee 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc 00007fe455e9e782 00007fe455ca2bcc 00007fe455ca2006
0501 20:01:42.221585 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2061084416}
I20240501 20:01:42.228472 29801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41864 (request call id 1005) took 1931 ms. Trace:
I20240501 20:01:42.228889 29801 rpcz_store.cc:276] 0501 20:01:40.296731 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:40.418221 (+121490us) service_pool.cc:225] Handling call
0501 20:01:40.418804 (+ 583us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:41.249850 (+831046us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:42.228247 (+978397us) spinlock_profiling.cc:243] Waited 823 ms on lock 0x7b540014f5d4. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787c1a1 00007fe45787abee 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc 00007fe455e9e782 00007fe455ca2bcc 00007fe455ca2006
0501 20:01:42.228414 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1892975360}
I20240501 20:01:42.225798 29699 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 812)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 939)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 951)
I20240501 20:01:42.230068 29801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458740ab5c444b6ab3d1991c69db5084" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:42.230918 29801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be383be783234855878cb8fd442c7b05" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:42.230623 29699 service_pool.cc:138] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 812)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 939)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 951)
I20240501 20:01:42.234371 29699 service_pool.cc:138] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 812)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 939)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 951)
I20240501 20:01:42.237972 29699 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 812)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 939)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 951)
I20240501 20:01:42.246694 29801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f5d1686214485d91a0baf4b870740e" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
W20240501 20:01:42.247735 29624 leader_election.cc:334] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 1.921s (SENT)
I20240501 20:01:42.247949 29801 raft_consensus.cc:3037] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Advancing to term 5
I20240501 20:01:42.249866 29654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1a08b455b94ae089396a15c837ddc2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:42.250437 29654 raft_consensus.cc:3037] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:42.251157 29657 raft_consensus.cc:2454] T 2d86f8848a184a87947fd927c44e5a35 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 4.
I20240501 20:01:42.251821 31186 raft_consensus.cc:3037] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:42.242655 31249 raft_consensus.cc:504] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:42.248961 29624 leader_election.cc:334] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Remote error: Service unavailable: RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items.
I20240501 20:01:42.199018 31232 leader_election.cc:288] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:42.255928 29624 leader_election.cc:334] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Remote error: Service unavailable: RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items.
I20240501 20:01:42.256425 29659 raft_consensus.cc:2454] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 5.
I20240501 20:01:42.259358 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668b4a4d782c4278932dd8413dc3ab3e" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
W20240501 20:01:42.256654 29624 consensus_peers.cc:485] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f -> Peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Couldn't send request to peer fdfeac4190e045f290e003b2298d988d. Status: Remote error: Service unavailable: UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items.. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:42.261507 29658 raft_consensus.cc:2379] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 4: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
W20240501 20:01:42.261600 29624 leader_election.cc:334] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Remote error: Service unavailable: RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items.
I20240501 20:01:42.245250 29802 raft_consensus.cc:1226] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 4. Current term is 5. Ops: []
I20240501 20:01:42.264086 31249 leader_election.cc:288] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:42.263265 29624 leader_election.cc:334] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Remote error: Service unavailable: RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items.
W20240501 20:01:42.266161 29624 consensus_peers.cc:485] T 010ceabf3e5a4b1c9330b80dade765bb P 4c8da1d580614449a636b855bf11b95f -> Peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Couldn't send request to peer fdfeac4190e045f290e003b2298d988d. Status: Remote error: Service unavailable: UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items.. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:42.268478 29772 leader_election.cc:302] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:42.269317 29772 leader_election.cc:302] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:42.270800 29804 raft_consensus.cc:2454] T 1ca3948261374cef8b1b206a43c00173 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 2.
I20240501 20:01:42.271600 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0374bb790ed345ed90452595f71383a2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:42.272318 29657 raft_consensus.cc:3037] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:42.272591 29700 leader_election.cc:302] T 1ca3948261374cef8b1b206a43c00173 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:42.274194 29659 raft_consensus.cc:1261] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240501 20:01:42.271637 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010ceabf3e5a4b1c9330b80dade765bb" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:42.266592 31171 consensus_queue.cc:1035] T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:42.277921 31232 raft_consensus.cc:2781] T 1ca3948261374cef8b1b206a43c00173 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:01:42.278352 31232 raft_consensus.cc:686] T 1ca3948261374cef8b1b206a43c00173 P fdfeac4190e045f290e003b2298d988d [term 2 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:42.279145 31232 consensus_queue.cc:237] T 1ca3948261374cef8b1b206a43c00173 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:42.281111 31152 raft_consensus.cc:2781] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:01:42.259405 31250 consensus_queue.cc:1046] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.015s
I20240501 20:01:42.282860 29651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a35893b91247a99912eba787d787cf" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:42.283521 29651 raft_consensus.cc:3037] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:42.284025 31250 raft_consensus.cc:3032] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:01:42.284175 30849 leader_election.cc:288] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:42.284446 31250 raft_consensus.cc:729] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:42.276633 29699 service_pool.cc:138] GetConsensusState request on kudu.consensus.ConsensusService from 127.0.0.1:36258 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 812)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 939)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 951)
I20240501 20:01:42.265918 31243 raft_consensus.cc:2781] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:01:42.281208 31019 raft_consensus.cc:2781] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:01:42.285390 31106 raft_consensus.cc:482] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240501 20:01:42.296012 29625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 481499us
W20240501 20:01:42.289695 29624 leader_election.cc:334] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 1.826s (SENT)
I20240501 20:01:42.292212 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5525f546426c47a38edf7f56ad04b7ab" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:42.297396 29658 raft_consensus.cc:3037] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:42.290473 29804 raft_consensus.cc:1226] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20240501 20:01:42.294660 30849 consensus_queue.cc:1035] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:42.296845 29801 raft_consensus.cc:2454] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 5.
I20240501 20:01:42.298646 29624 leader_election.cc:302] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:42.300156 29700 leader_election.cc:302] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:42.287017 31152 raft_consensus.cc:686] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [term 4 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:42.305245 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcf30cf66714efdac997862dafbe4a2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:42.307895 29801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07bdb8b09b8c4ef4b8e49daa9b8c01d6" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:42.311120 29801 raft_consensus.cc:1226] T 1ca3948261374cef8b1b206a43c00173 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:42.304603 29699 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 812)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 939)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 951)
I20240501 20:01:42.317257 29657 raft_consensus.cc:2454] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 4.
I20240501 20:01:42.320948 29657 raft_consensus.cc:1261] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20240501 20:01:42.321844 29624 leader_election.cc:334] T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 2.068s (SENT)
I20240501 20:01:42.323055 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8787bc4d6b44649b9138d906730110a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:42.316653 29699 service_pool.cc:138] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 812)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 939)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 951)
I20240501 20:01:42.325889 29699 service_pool.cc:138] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 812)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 939)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 951)
I20240501 20:01:42.301733 31243 raft_consensus.cc:686] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d [term 4 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:42.303447 31249 raft_consensus.cc:2726] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:01:42.301523 31106 raft_consensus.cc:504] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:42.347258 31153 leader_election.cc:288] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:42.348274 31150 leader_election.cc:288] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:42.349136 29728 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 823) took 1510 ms (client timeout 1517 ms). Trace:
I20240501 20:01:42.348939 31101 consensus_queue.cc:1046] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:42.352205 29656 raft_consensus.cc:595] T 1ca3948261374cef8b1b206a43c00173 P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Received request to transfer leadership to TS fdfeac4190e045f290e003b2298d988d
W20240501 20:01:42.352929 30417 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40670 (request call id 47) took 1748 ms (client timeout 1946 ms). Trace:
I20240501 20:01:42.346231 31135 leader_election.cc:288] T a8787bc4d6b44649b9138d906730110a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
W20240501 20:01:42.359320 31190 log.cc:924] Time spent T 00000000000000000000000000000000 P 995802798c1e438daecd368538bebecd: Append to log took a long time: real 2.085s user 0.001s sys 0.001s
I20240501 20:01:42.293660 31250 consensus_queue.cc:260] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:42.356334 29699 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 812)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 939)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 951)
I20240501 20:01:42.319125 31063 raft_consensus.cc:482] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:42.389477 29699 service_pool.cc:138] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 812)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 939)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 959)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60)
I20240501 20:01:42.328330 29658 raft_consensus.cc:2454] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 5.
I20240501 20:01:42.398708 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb27193abbf4644a48662909bc7b04a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:42.399619 29658 raft_consensus.cc:2379] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d in current term 4: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
W20240501 20:01:42.310699 30384 leader_election.cc:334] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Remote error: Service unavailable: RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:47414 dropped due to backpressure. The service queue is full; it has 50 items.
I20240501 20:01:42.319656 31019 raft_consensus.cc:686] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 5 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:42.344090 31246 raft_consensus.cc:2781] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:01:42.386413 29804 raft_consensus.cc:2454] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 5.
I20240501 20:01:42.386750 29698 leader_election.cc:302] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, 888d99522e4844118f8f1cf26f268c8c
I20240501 20:01:42.392575 29656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:60736 (request call id 0) took 1218 ms. Trace:
I20240501 20:01:42.384640 31243 consensus_queue.cc:237] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:42.393066 31106 leader_election.cc:288] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:42.384402 30417 rpcz_store.cc:269] 0501 20:01:40.619335 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:40.621371 (+ 2036us) service_pool.cc:225] Handling call
0501 20:01:42.352767 (+1731396us) spinlock_profiling.cc:243] Waited 40.2 ms on lock 0x7b540044ff54. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787f4fe 00007fe45c7bd9d7 00007fe455ea1c14 00007fe455ea1ba2 00007fe455ea1ad1 00007fe455ea1a4c 00007fe455ea0d02 00007fe455ca2bcc 00007fe455ca2006 00007fe455ca19af
0501 20:01:42.352859 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":92529408}
I20240501 20:01:42.398413 29699 service_pool.cc:138] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 904)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 812)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 939)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 959)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60)
I20240501 20:01:42.370801 29654 raft_consensus.cc:2454] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 4.
I20240501 20:01:42.423471 31250 raft_consensus.cc:3037] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:42.425406 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f5d1686214485d91a0baf4b870740e" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:42.439216 29658 raft_consensus.cc:3037] T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:42.427368 31221 raft_consensus.cc:504] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:42.429167 31083 consensus_queue.cc:1046] T 3c5345f6eb62449ba69b4a9c2e349b31 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 20:01:42.405925 31150 raft_consensus.cc:482] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:42.430102 29731 raft_consensus.cc:2396] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f for term 5 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20240501 20:01:42.430153 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.196:37227, user_credentials={real_user=slave}} blocked reactor thread for 43513us
I20240501 20:01:42.445217 29698 leader_election.cc:302] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, 888d99522e4844118f8f1cf26f268c8c
I20240501 20:01:42.446198 29698 leader_election.cc:302] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, 888d99522e4844118f8f1cf26f268c8c
W20240501 20:01:42.382879 29728 rpcz_store.cc:269] 0501 20:01:40.838901 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:40.910702 (+ 71801us) service_pool.cc:225] Handling call
0501 20:01:42.348992 (+1438290us) spinlock_profiling.cc:243] Waited 141 ms on lock 0x7b5400230fd4. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787f4fe 00007fe45c7bd9d7 00007fe455ea1c14 00007fe455ea1ba2 00007fe455ea1ad1 00007fe455ea1a4c 00007fe455ea0d02 00007fe455ca2bcc 00007fe455ca2006 00007fe455ca19af
0501 20:01:42.349086 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":323831552}
I20240501 20:01:42.447398 31020 leader_election.cc:288] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:42.421011 29656 rpcz_store.cc:276] 0501 20:01:41.165691 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:41.165832 (+ 141us) service_pool.cc:225] Handling call
0501 20:01:42.352059 (+1186227us) spinlock_profiling.cc:243] Waited 4.08 ms on lock 0x7b5800001598. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe454797b51 00007fe4549193b1 00007fe454919262 00007fe45d80b889 00007fe45c7483c0 00007fe45c80e3bb 00007fe45c80e441 00007fe45c7d50e0 00007fe45c7c0b79 00007fe45c7c043c 00007fe455eafd14
0501 20:01:42.384182 (+ 32123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9388288}
I20240501 20:01:42.448258 31101 raft_consensus.cc:2726] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:42.448536 31150 raft_consensus.cc:504] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:42.449065 31101 raft_consensus.cc:2726] T 7ed27033ab1645e9ab0600ab635d9589 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240501 20:01:42.449020 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d2cac25f354a97b1bfb53d57ae7c08" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:42.450196 29656 raft_consensus.cc:3037] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:42.450625 31150 leader_election.cc:288] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:42.451184 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458740ab5c444b6ab3d1991c69db5084" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:42.451462 29651 raft_consensus.cc:2454] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 4.
I20240501 20:01:42.451628 31083 raft_consensus.cc:3032] T 3c5345f6eb62449ba69b4a9c2e349b31 P 4c8da1d580614449a636b855bf11b95f [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:01:42.452066 31083 raft_consensus.cc:729] T 3c5345f6eb62449ba69b4a9c2e349b31 P 4c8da1d580614449a636b855bf11b95f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:42.449060 29654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36057b3f24d44b9aac3a1378053a395f" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:42.453696 29654 raft_consensus.cc:1226] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20240501 20:01:42.453480 31083 consensus_queue.cc:260] T 3c5345f6eb62449ba69b4a9c2e349b31 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:42.455150 29653 raft_consensus.cc:1226] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 3. Current term is 4. Ops: []
I20240501 20:01:42.412194 29772 leader_election.cc:302] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
W20240501 20:01:42.437268 29699 service_pool.cc:135] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items. [suppressed 36 similar messages]
I20240501 20:01:42.455716 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c651044a305c4168805825a53d910899" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:42.440883 31246 raft_consensus.cc:686] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [term 5 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:42.396071 31152 consensus_queue.cc:237] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:42.438683 31186 raft_consensus.cc:2726] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240501 20:01:42.436899 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb27193abbf4644a48662909bc7b04a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:42.463276 29804 raft_consensus.cc:2379] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 4: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:42.464116 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668b4a4d782c4278932dd8413dc3ab3e" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:42.465193 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af08384057a4d85977481472f460168" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:42.465829 29658 raft_consensus.cc:2454] T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 5.
I20240501 20:01:42.408358 31063 raft_consensus.cc:504] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:42.467514 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6797d82dda2948d8a7445fcd33e44752" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:42.456990 29772 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 44880.2us
I20240501 20:01:42.448204 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31fff64b453a423eb0f567b3171564fb" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:42.448164 31220 raft_consensus.cc:482] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:42.462007 29660 raft_consensus.cc:1226] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 4. Current term is 5. Ops: [2.2-4.3]
I20240501 20:01:42.469606 29658 raft_consensus.cc:1226] T 2d86f8848a184a87947fd927c44e5a35 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20240501 20:01:42.461220 31181 raft_consensus.cc:482] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:42.448030 29698 service_pool.cc:138] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:36260 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 812)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 939)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 959)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60)
I20240501 20:01:42.479066 29657 raft_consensus.cc:1261] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240501 20:01:42.462860 31083 raft_consensus.cc:3037] T 3c5345f6eb62449ba69b4a9c2e349b31 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:42.465663 31214 raft_consensus.cc:2781] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:01:42.477730 29772 leader_election.cc:302] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:42.411089 30384 leader_election.cc:302] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 888d99522e4844118f8f1cf26f268c8c; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:42.479661 29698 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36260 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 812)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 939)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 959)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 827)
I20240501 20:01:42.486212 29772 leader_election.cc:302] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:42.465193 31246 consensus_queue.cc:237] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:42.488269 31279 consensus_queue.cc:1046] T 1ca3948261374cef8b1b206a43c00173 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20240501 20:01:42.489332 30849 raft_consensus.cc:2781] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:01:42.492834 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfac893b4b4f457694a4eb184b6abb53" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:42.512348 29655 raft_consensus.cc:3037] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:42.505087 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e38a2e91a3d48d09a22abce723af833" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:42.513561 29804 raft_consensus.cc:2379] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 4: Already voted for candidate ee0d97247d394a8facfa774f15f3a67d in this term.
I20240501 20:01:42.514441 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6797d82dda2948d8a7445fcd33e44752" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:42.509543 29658 raft_consensus.cc:1261] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240501 20:01:42.510808 29698 service_pool.cc:138] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:36260 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 812)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 939)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 959)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 828)
I20240501 20:01:42.529367 29698 service_pool.cc:138] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:36260 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 812)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 939)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 959)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 828)
I20240501 20:01:42.507761 29651 raft_consensus.cc:1226] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20240501 20:01:42.504324 31186 raft_consensus.cc:482] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:42.547537 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77be7101c0fe4085b6706296f609870a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:42.547747 31186 raft_consensus.cc:504] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:42.548844 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac2171683884f6f97cdfeb0a656cb49" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:42.549996 31186 leader_election.cc:288] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:42.543128 29772 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 57034.5us
I20240501 20:01:42.480130 31020 consensus_queue.cc:1035] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.035s
I20240501 20:01:42.546682 30849 raft_consensus.cc:686] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [term 5 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:42.551753 29772 leader_election.cc:302] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:42.478113 31220 raft_consensus.cc:504] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:42.507261 31214 raft_consensus.cc:686] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 4 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:42.556094 29625 leader_election.cc:302] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:42.562254 31206 raft_consensus.cc:2781] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:01:42.563274 29625 leader_election.cc:302] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:42.563330 31206 raft_consensus.cc:686] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d [term 4 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:42.564383 31206 consensus_queue.cc:237] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:42.565109 31140 raft_consensus.cc:2781] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:01:42.565542 31140 raft_consensus.cc:686] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [term 5 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:42.566252 31140 consensus_queue.cc:237] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:42.566998 30849 consensus_queue.cc:237] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:42.513411 29654 raft_consensus.cc:1226] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 4. Current term is 5. Ops: []
I20240501 20:01:42.560176 31237 raft_consensus.cc:2726] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240501 20:01:42.485169 31181 raft_consensus.cc:504] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:42.512677 29728 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 904) took 1572 ms (client timeout 1830 ms). Trace:
I20240501 20:01:42.551788 31135 raft_consensus.cc:2781] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:01:42.572568 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:42.578702 29656 raft_consensus.cc:2454] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 4.
I20240501 20:01:42.582480 31268 raft_consensus.cc:2726] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:42.584399 29658 raft_consensus.cc:2454] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 3.
I20240501 20:01:42.586138 31135 raft_consensus.cc:686] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d [term 4 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:42.587399 29800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41864 (request call id 1014) took 1963 ms. Trace:
I20240501 20:01:42.587872 29800 rpcz_store.cc:276] 0501 20:01:40.623921 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:41.530907 (+906986us) service_pool.cc:225] Handling call
0501 20:01:41.531276 (+ 369us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:41.620617 (+ 89341us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:42.587219 (+966602us) spinlock_profiling.cc:243] Waited 67.6 ms on lock 0x7b54002ef254. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787c1a1 00007fe45787abee 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc 00007fe455e9e782 00007fe455ca2bcc 00007fe455ca2006
0501 20:01:42.587352 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":155488512}
I20240501 20:01:42.589141 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece8d4ed84c34b748f7b6272b31c90ac" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:42.494920 31019 consensus_queue.cc:237] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:42.560757 29651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649b71f9f267437ca51c758dd9ca6987" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:42.592620 31181 leader_election.cc:288] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:42.593065 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a2bba62f434d1c8489b55c8fec2187" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:42.455965 29699 service_pool.cc:138] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 812)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 939)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 959)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60)
W20240501 20:01:42.581389 29728 rpcz_store.cc:269] 0501 20:01:40.938302 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:42.447886 (+1509584us) service_pool.cc:225] Handling call
0501 20:01:42.510715 (+ 62829us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240501 20:01:42.596499 29735 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 813) took 1800 ms (client timeout 1857 ms). Trace:
W20240501 20:01:42.625994 29735 rpcz_store.cc:269] 0501 20:01:40.796631 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:41.509481 (+712850us) service_pool.cc:225] Handling call
0501 20:01:42.596336 (+1086855us) spinlock_profiling.cc:243] Waited 210 ms on lock 0x7b54002d87d4. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787f4fe 00007fe45c7bd9d7 00007fe455ea1c14 00007fe455ea1ba2 00007fe455ea1ad1 00007fe455ea1a4c 00007fe455ea0d02 00007fe455ca2bcc 00007fe455ca2006 00007fe455ca19af
0501 20:01:42.596451 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":483861760}
I20240501 20:01:42.628751 29808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41864 (request call id 1028) took 1256 ms. Trace:
I20240501 20:01:42.629159 29808 rpcz_store.cc:276] 0501 20:01:41.372280 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:41.759064 (+386784us) service_pool.cc:225] Handling call
0501 20:01:41.931346 (+172282us) spinlock_profiling.cc:243] Waited 17.2 ms on lock 0x7b58000d2c98. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe454797b51 00007fe4549193b1 00007fe454919262 00007fe45d80b889 00007fe45c7483c0 00007fe45c80e3bb 00007fe45c80e441 00007fe45c7d2060 00007fe45c7bd5a9 00007fe45c7bd052 00007fe455e9f694
0501 20:01:41.931555 (+ 209us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:42.050903 (+119348us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:42.628460 (+577557us) spinlock_profiling.cc:243] Waited 117 ms on lock 0x7b5400148cd4. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787c1a1 00007fe45787abee 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc 00007fe455e9e782 00007fe455ca2bcc 00007fe455ca2006
0501 20:01:42.628687 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":308531200}
I20240501 20:01:42.620616 29806 raft_consensus.cc:1226] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [term 4 LEADER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20240501 20:01:42.633520 31135 consensus_queue.cc:237] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:42.570855 31165 consensus_queue.cc:1035] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.026s
I20240501 20:01:42.596987 31221 leader_election.cc:288] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:42.622895 29729 raft_consensus.cc:2454] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 3.
I20240501 20:01:42.624296 31214 consensus_queue.cc:237] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:42.708578 31079 raft_consensus.cc:482] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:42.644692 29655 raft_consensus.cc:2454] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 5.
I20240501 20:01:42.707574 31132 ts_tablet_manager.cc:1888] T be383be783234855878cb8fd442c7b05 P 888d99522e4844118f8f1cf26f268c8c: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.3
I20240501 20:01:42.710501 29651 raft_consensus.cc:3037] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:42.555083 29698 service_pool.cc:138] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:36260 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 820)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 812)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 939)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 959)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 828)
I20240501 20:01:42.835836 29557 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:43370 (request call id 154) took 7102 ms. Trace:
W20240501 20:01:42.713671 29729 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 817) took 1902 ms (client timeout 1884 ms). Trace:
W20240501 20:01:42.731268 30385 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.597s user 0.020s sys 0.044s
I20240501 20:01:43.120586 29806 raft_consensus.cc:1261] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240501 20:01:43.121656 29800 raft_consensus.cc:1261] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20240501 20:01:42.916568 29773 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.657s user 0.000s sys 0.035s
W20240501 20:01:43.151638 29773 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.892s user 0.000s sys 0.044s
W20240501 20:01:43.148336 30385 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.014s user 0.020s sys 0.049s
W20240501 20:01:43.151996 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 892961us
I20240501 20:01:43.154107 30385 leader_election.cc:398] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 4 pre-election: Vote denied by peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861) with higher term. Message: Invalid argument: T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 888d99522e4844118f8f1cf26f268c8c for earlier term 4. Current term is 5.
I20240501 20:01:43.111249 31132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be383be783234855878cb8fd442c7b05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:43.120911 29698 service_pool.cc:138] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:36260 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 939)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 959)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 828)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 833)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 831)
I20240501 20:01:43.157317 29800 raft_consensus.cc:1261] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Refusing update from remote peer 888d99522e4844118f8f1cf26f268c8c: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20240501 20:01:43.124748 29624 leader_election.cc:334] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 1.622s (SENT)
I20240501 20:01:43.125368 29625 leader_election.cc:302] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:43.154456 30385 leader_election.cc:401] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20240501 20:01:43.122787 29772 leader_election.cc:302] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:43.155233 31279 raft_consensus.cc:482] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 888d99522e4844118f8f1cf26f268c8c)
W20240501 20:01:43.117748 31083 consensus_meta.cc:287] T 3c5345f6eb62449ba69b4a9c2e349b31 P 4c8da1d580614449a636b855bf11b95f: Time spent flushing consensus metadata: real 0.634s user 0.009s sys 0.000s
I20240501 20:01:43.161257 29625 leader_election.cc:302] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:43.169584 29773 leader_election.cc:302] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
W20240501 20:01:42.917227 31267 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.968s user 0.013s sys 0.020s
I20240501 20:01:43.157795 29698 service_pool.cc:138] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:36260 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 939)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 959)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 828)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 833)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 831)
I20240501 20:01:43.178510 31109 raft_consensus.cc:2726] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 20:01:43.191483 29819 tablet_copy_service.cc:342] P ee0d97247d394a8facfa774f15f3a67d: Request end of tablet copy session 888d99522e4844118f8f1cf26f268c8c-77be7101c0fe4085b6706296f609870a received from {username='slave'} at 127.0.0.1:49474
I20240501 20:01:43.191916 29819 tablet_copy_service.cc:433] P ee0d97247d394a8facfa774f15f3a67d: ending tablet copy session 888d99522e4844118f8f1cf26f268c8c-77be7101c0fe4085b6706296f609870a on tablet 77be7101c0fe4085b6706296f609870a with peer 888d99522e4844118f8f1cf26f268c8c
I20240501 20:01:43.160356 29624 leader_election.cc:302] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: 888d99522e4844118f8f1cf26f268c8c, fdfeac4190e045f290e003b2298d988d
W20240501 20:01:42.920749 30944 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.944s user 0.016s sys 0.016s
W20240501 20:01:42.917690 31253 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.983s user 0.011s sys 0.028s
I20240501 20:01:43.196614 29773 leader_election.cc:302] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 888d99522e4844118f8f1cf26f268c8c, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:43.164788 31020 raft_consensus.cc:482] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:43.163025 31153 raft_consensus.cc:2726] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20240501 20:01:43.171483 31267 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.224s user 0.013s sys 0.026s
I20240501 20:01:42.736132 31079 raft_consensus.cc:504] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:42.843340 29557 rpcz_store.cc:276] 0501 20:01:35.733503 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:35.745633 (+ 12130us) service_pool.cc:225] Handling call
0501 20:01:42.835713 (+7090080us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0501 20:01:37.356147 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 20:01:37.379857 (+ 23710us) write_op.cc:424] Acquiring schema lock in shared mode
0501 20:01:37.382013 (+ 2156us) write_op.cc:427] Acquired schema lock
0501 20:01:37.382033 (+ 20us) tablet.cc:660] Decoding operations
0501 20:01:37.383236 (+ 1203us) write_op.cc:612] Acquiring the partition lock for write op
0501 20:01:37.383309 (+ 73us) write_op.cc:633] Partition lock acquired for write op
0501 20:01:37.383342 (+ 33us) tablet.cc:683] Acquiring locks for 12 operations
0501 20:01:37.388303 (+ 4961us) tablet.cc:699] Row locks acquired
0501 20:01:37.388323 (+ 20us) write_op.cc:260] PREPARE: finished
0501 20:01:37.388476 (+ 153us) write_op.cc:270] Start()
0501 20:01:37.388589 (+ 113us) write_op.cc:276] Timestamp: P: 1714593697388430 usec, L: 0
0501 20:01:37.388615 (+ 26us) op_driver.cc:331] REPLICATION: starting
0501 20:01:37.389259 (+ 644us) log.cc:842] Serialized 4845 byte log entry
0501 20:01:37.609749 (+220490us) op_driver.cc:444] REPLICATION: finished
0501 20:01:37.704845 (+ 95096us) write_op.cc:301] APPLY: starting
0501 20:01:37.714810 (+ 9965us) tablet.cc:1365] starting BulkCheckPresence
0501 20:01:37.806807 (+ 91997us) tablet.cc:1368] finished BulkCheckPresence
0501 20:01:37.951276 (+144469us) tablet.cc:1370] starting ApplyRowOperation cycle
0501 20:01:38.059148 (+107872us) tablet.cc:1381] finished ApplyRowOperation cycle
0501 20:01:38.059374 (+ 226us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=12
0501 20:01:38.059420 (+ 46us) write_op.cc:312] APPLY: finished
0501 20:01:38.059706 (+ 286us) log.cc:842] Serialized 88 byte log entry
0501 20:01:38.059870 (+ 164us) write_op.cc:481] Releasing partition, row and schema locks
0501 20:01:38.060008 (+ 138us) write_op.cc:446] Released schema lock
0501 20:01:38.060082 (+ 74us) write_op.cc:341] FINISH: Updating metrics
0501 20:01:38.060340 (+ 258us) write_op.cc:481] Releasing partition, row and schema locks
0501 20:01:38.060365 (+ 25us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":42439,"reported_tablets":29,"child_traces":[["op",{"apply.queue_time_us":59046,"apply.run_cpu_time_us":12952,"apply.run_wall_time_us":383853,"num_ops":12,"prepare.queue_time_us":166224,"prepare.run_cpu_time_us":4902,"prepare.run_wall_time_us":74966,"replication_time_us":213229,"thread_start_us":70133,"threads_started":2}]]}
W20240501 20:01:43.194744 29772 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 72040.7us
W20240501 20:01:43.195284 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 70575us
W20240501 20:01:43.201642 29659 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.838s user 0.021s sys 0.026s
W20240501 20:01:43.220916 29659 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.857s user 0.022s sys 0.026s
W20240501 20:01:43.127851 30384 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 827395us
I20240501 20:01:43.163838 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "696121417ea9403990456cfc6fc4f416" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:43.209380 29733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 880) took 2605 ms. Trace:
I20240501 20:01:43.221928 29733 rpcz_store.cc:276] 0501 20:01:40.602486 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:40.605576 (+ 3090us) service_pool.cc:225] Handling call
0501 20:01:43.204590 (+2599014us) spinlock_profiling.cc:243] Waited 2.51 ms on lock 0x7b58001b3098. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe454797b51 00007fe4549193b1 00007fe454919262 00007fe45d80b889 00007fe45c7483c0 00007fe45c80e3bb 00007fe45c80e441 00007fe45c7d2060 00007fe45c7bd5a9 00007fe45c7bd052 00007fe455e9f694
0501 20:01:43.204824 (+ 234us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:43.205011 (+ 187us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:43.205053 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:43.205098 (+ 45us) raft_consensus.cc:1631] Marking committed up to 1
0501 20:01:43.205133 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:43.205190 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:43.205292 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5773056}
W20240501 20:01:43.209851 29801 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.850s user 0.032s sys 0.035s
W20240501 20:01:43.196202 30944 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.219s user 0.019s sys 0.019s
W20240501 20:01:43.196372 31253 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.261s user 0.011s sys 0.035s
I20240501 20:01:43.198180 29698 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36260 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 798)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 939)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 959)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 828)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 833)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 831)
W20240501 20:01:43.148524 29729 rpcz_store.cc:269] 0501 20:01:40.811037 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:41.614366 (+803329us) service_pool.cc:225] Handling call
0501 20:01:42.592916 (+978550us) spinlock_profiling.cc:243] Waited 1.49 ms on lock 0x7b58001b3098. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe454797b51 00007fe4549193b1 00007fe454919262 00007fe45d80b889 00007fe45c7483c0 00007fe45c80e3bb 00007fe45c80e441 00007fe45c7bdf60 00007fe45c7bd844 00007fe455ea1c14 00007fe455ea1ba2
0501 20:01:42.712134 (+119218us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3429632}
I20240501 20:01:43.195936 31268 raft_consensus.cc:2726] T 7ed27033ab1645e9ab0600ab635d9589 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20240501 20:01:43.224634 29801 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.865s user 0.032s sys 0.035s
W20240501 20:01:43.230957 29733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 798) took 2529 ms (client timeout 2102 ms). Trace:
I20240501 20:01:43.171051 31279 raft_consensus.cc:504] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:43.229027 29698 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36260 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 939)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 959)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 828)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 833)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 831)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 837)
I20240501 20:01:43.295238 29698 service_pool.cc:138] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:36260 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 939)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 959)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 828)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 833)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 831)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 838)
I20240501 20:01:43.206741 31178 raft_consensus.cc:2726] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240501 20:01:43.206558 31221 raft_consensus.cc:2726] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:01:43.208827 31197 raft_consensus.cc:2781] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election won for term 5
W20240501 20:01:43.256242 29733 rpcz_store.cc:269] 0501 20:01:40.701090 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:43.228651 (+2527561us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:43.230898 (+ 2247us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240501 20:01:43.207314 31020 raft_consensus.cc:504] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:43.291162 31292 raft_consensus.cc:3037] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Advancing to term 5
I20240501 20:01:43.299635 29698 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36260 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 939)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 959)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 828)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 833)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 831)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 838)
I20240501 20:01:43.212728 31063 leader_election.cc:288] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:43.334041 29733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 806) took 2630 ms (client timeout 2123 ms). Trace:
W20240501 20:01:43.336814 29733 rpcz_store.cc:269] 0501 20:01:40.703389 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:43.333886 (+2630497us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:43.333989 (+ 103us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240501 20:01:43.339854 29733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 911) took 2368 ms (client timeout 1865 ms). Trace:
W20240501 20:01:43.340234 29733 rpcz_store.cc:269] 0501 20:01:40.969198 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:43.337514 (+2368316us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:43.337643 (+ 129us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240501 20:01:43.341099 29733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 808) took 2637 ms (client timeout 2145 ms). Trace:
W20240501 20:01:43.341454 29733 rpcz_store.cc:269] 0501 20:01:40.703922 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:43.340880 (+2636958us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:43.340994 (+ 114us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240501 20:01:43.342286 29733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 822) took 2512 ms (client timeout 2033 ms). Trace:
W20240501 20:01:43.342697 29733 rpcz_store.cc:269] 0501 20:01:40.829828 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:43.342120 (+2512292us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:43.342239 (+ 119us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240501 20:01:43.328966 31197 raft_consensus.cc:482] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240501 20:01:43.345075 29733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 893) took 2718 ms (client timeout 2246 ms). Trace:
I20240501 20:01:43.322777 31267 raft_consensus.cc:482] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 888d99522e4844118f8f1cf26f268c8c)
W20240501 20:01:43.346192 29733 rpcz_store.cc:269] 0501 20:01:40.626967 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:43.344927 (+2717960us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:43.345028 (+ 101us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240501 20:01:43.341872 29729 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 914) took 2240 ms (client timeout 1722 ms). Trace:
I20240501 20:01:43.334157 29698 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:36260 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 939)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 959)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 828)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 833)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 831)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 842)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 841)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 838)
I20240501 20:01:43.396226 29809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41864 (request call id 996) took 3202 ms. Trace:
I20240501 20:01:43.348850 31197 raft_consensus.cc:3037] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:43.396852 29809 rpcz_store.cc:276] 0501 20:01:40.193565 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:40.193693 (+ 128us) service_pool.cc:225] Handling call
0501 20:01:40.193968 (+ 275us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:40.429375 (+235407us) raft_consensus.cc:1520] Early marking committed up to index 2
0501 20:01:40.442445 (+ 13070us) log.cc:842] Serialized 14 byte log entry
0501 20:01:40.474604 (+ 32159us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:40.474666 (+ 62us) raft_consensus.cc:1631] Marking committed up to 2
0501 20:01:40.474713 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:41.506658 (+1031945us) spinlock_profiling.cc:243] Waited 21.9 ms on lock 0x7b44000232cc. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe454797b51 00007fe454865685 00007fe4548660c2 00007fe45750e441 00007fe457509d4a 00007fe4574fdb1e 00007fe4574fce66 00007fe4574fca88 00007fe457506481 00007fe457506439 00007fe4575063c9
0501 20:01:43.395917 (+1889259us) spinlock_profiling.cc:243] Waited 235 ms on lock 0x7b540018f454. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787c1a1 00007fe45787abee 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc 00007fe455e9e782 00007fe455ca2bcc 00007fe455ca2006
0501 20:01:43.395986 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:43.396158 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1473,"prepare.queue_time_us":12341,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":21950,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":87,"spinlock_wait_cycles":591172224,"thread_start_us":31821,"threads_started":1,"wal-append.queue_time_us":32167,"wal-append.run_cpu_time_us":11854,"wal-append.run_wall_time_us":2121399}
W20240501 20:01:43.360450 31232 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.950s user 0.025s sys 0.059s
W20240501 20:01:43.429605 31232 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.020s user 0.025s sys 0.060s
I20240501 20:01:43.346099 31267 raft_consensus.cc:504] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:43.468384 31079 leader_election.cc:288] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:43.469996 31267 leader_election.cc:288] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:43.470880 29659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59136 (request call id 818) took 1168 ms. Trace:
I20240501 20:01:43.471320 29659 rpcz_store.cc:276] 0501 20:01:42.302437 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:42.303069 (+ 632us) service_pool.cc:225] Handling call
0501 20:01:42.319486 (+ 16417us) raft_consensus.cc:1457] Updating replica for 2 ops
0501 20:01:42.319805 (+ 319us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:42.319833 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0501 20:01:42.320273 (+ 440us) log.cc:842] Serialized 52 byte log entry
0501 20:01:43.221379 (+901106us) raft_consensus.cc:1631] Marking committed up to 1
0501 20:01:43.221428 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:43.221510 (+ 82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0501 20:01:43.470634 (+249124us) raft_consensus.cc:1671] finished
0501 20:01:43.470657 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:43.470819 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1720,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":18976,"thread_start_us":857850,"threads_started":1,"wal-append.queue_time_us":996087}
W20240501 20:01:43.479768 29698 service_pool.cc:135] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:36260 dropped due to backpressure. The service queue is full; it has 50 items. [suppressed 12 similar messages]
I20240501 20:01:43.488998 31279 leader_election.cc:288] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
W20240501 20:01:43.389258 29729 rpcz_store.cc:269] 0501 20:01:41.100738 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:43.333303 (+2232565us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:43.340788 (+ 7485us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240501 20:01:43.492427 30650 leader_election.cc:288] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:43.495136 29807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41864 (request call id 1023) took 2170 ms. Trace:
I20240501 20:01:43.495587 29807 rpcz_store.cc:276] 0501 20:01:41.324439 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:41.749624 (+425185us) service_pool.cc:225] Handling call
0501 20:01:41.749931 (+ 307us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:41.814785 (+ 64854us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:41.814804 (+ 19us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:41.814828 (+ 24us) raft_consensus.cc:1631] Marking committed up to 2
0501 20:01:41.814844 (+ 16us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:43.494824 (+1679980us) spinlock_profiling.cc:243] Waited 64.7 ms on lock 0x7b5400211454. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787c1a1 00007fe45787abee 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc 00007fe455e9e782 00007fe455ca2bcc 00007fe455ca2006
0501 20:01:43.494888 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:43.495069 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":148738816}
W20240501 20:01:43.456238 29651 consensus_meta.cc:287] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f: Time spent flushing consensus metadata: real 0.606s user 0.006s sys 0.004s
I20240501 20:01:43.455835 31153 raft_consensus.cc:482] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:43.501394 31020 leader_election.cc:288] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:43.499248 29651 raft_consensus.cc:2454] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 5.
I20240501 20:01:43.502436 31153 raft_consensus.cc:504] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:43.490679 31206 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.922s user 0.006s sys 0.017s
W20240501 20:01:43.526845 29735 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 820) took 2710 ms (client timeout 1954 ms). Trace:
W20240501 20:01:43.527364 29735 rpcz_store.cc:269] 0501 20:01:40.816496 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:43.526615 (+2710119us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:43.526753 (+ 138us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240501 20:01:43.522437 29802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41864 (request call id 1018) took 2199 ms. Trace:
W20240501 20:01:43.526212 31206 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.958s user 0.006s sys 0.017s
I20240501 20:01:43.484450 29698 service_pool.cc:138] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:36260 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 939)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 959)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 828)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 833)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 831)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 846)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 842)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 841)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 848)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 843)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 838)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 845)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 847)
W20240501 20:01:43.530745 29804 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.980s user 0.010s sys 0.015s
W20240501 20:01:43.530088 31190 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.099s user 0.102s sys 0.124s
W20240501 20:01:43.535514 29775 thread.cc:636] server-negotiator [worker] (thread pool) Time spent creating pthread: real 1.026s user 0.009s sys 0.016s
W20240501 20:01:43.537545 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 340963us
I20240501 20:01:43.557662 29773 leader_election.cc:302] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
W20240501 20:01:43.531819 29804 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.981s user 0.010s sys 0.015s
W20240501 20:01:43.544984 29775 thread.cc:602] server-negotiator [worker] (thread pool) Time spent starting thread: real 1.036s user 0.009s sys 0.016s
W20240501 20:01:43.544440 31152 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.052s user 0.017s sys 0.010s
W20240501 20:01:43.576157 29660 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.989s user 0.011s sys 0.015s
W20240501 20:01:43.538122 31190 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.107s user 0.102s sys 0.124s
W20240501 20:01:43.538336 31243 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.085s user 0.021s sys 0.061s
W20240501 20:01:43.579732 29660 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.993s user 0.011s sys 0.015s
W20240501 20:01:43.574452 29774 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.718s user 0.014s sys 0.008s
W20240501 20:01:43.591352 29774 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.735s user 0.014s sys 0.009s
W20240501 20:01:43.576591 30849 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.996s user 0.014s sys 0.009s
I20240501 20:01:43.611407 30650 raft_consensus.cc:482] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 888d99522e4844118f8f1cf26f268c8c)
I20240501 20:01:43.621153 30650 raft_consensus.cc:504] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
W20240501 20:01:43.537067 29609 thread.cc:636] client-negotiator [worker] (thread pool) Time spent creating pthread: real 1.072s user 0.010s sys 0.016s
I20240501 20:01:43.619139 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9472c53f79ec423f82c396bd980674e2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:43.620318 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84fa28fb55f44b61ba1a88ed775a78ba" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:43.581449 31152 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.089s user 0.017s sys 0.010s
W20240501 20:01:43.589830 31243 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.144s user 0.021s sys 0.061s
W20240501 20:01:43.581660 31135 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.538s user 0.012s sys 0.011s
W20240501 20:01:43.625115 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 67518.9us
I20240501 20:01:43.631196 30650 leader_election.cc:288] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:43.635387 29808 raft_consensus.cc:1226] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [term 4 LEADER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 3. Current term is 4. Ops: []
W20240501 20:01:43.637046 29773 connection.cc:232] RPC call timeout handler was delayed by -1.3343410180000319s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20800000000000002s for a response
W20240501 20:01:43.637725 29773 connection.cc:232] RPC call timeout handler was delayed by -1.2956410180001967s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21230000000000004s for a response
W20240501 20:01:43.638057 29773 connection.cc:232] RPC call timeout handler was delayed by -1.2758410180001647s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21450000000000002s for a response
W20240501 20:01:43.638912 29773 connection.cc:232] RPC call timeout handler was delayed by -1.234137167000199s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2129s for a response
W20240501 20:01:43.639312 29773 leader_election.cc:334] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 1.517s (SENT)
W20240501 20:01:43.639850 29773 connection.cc:232] RPC call timeout handler was delayed by -1.1960941230001936s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19540000000000002s for a response
I20240501 20:01:43.530731 29802 rpcz_store.cc:276] 0501 20:01:41.322841 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:41.750540 (+427699us) service_pool.cc:225] Handling call
0501 20:01:41.784246 (+ 33706us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:42.263891 (+479645us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:43.522120 (+1258229us) spinlock_profiling.cc:243] Waited 424 ms on lock 0x7b540036d154. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787c1a1 00007fe45787abee 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc 00007fe455e9e782 00007fe455ca2bcc 00007fe455ca2006
0501 20:01:43.522362 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":975501184}
W20240501 20:01:43.640326 29773 leader_election.cc:334] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 1.857s (SENT)
W20240501 20:01:43.623291 29609 thread.cc:602] client-negotiator [worker] (thread pool) Time spent starting thread: real 1.158s user 0.010s sys 0.017s
I20240501 20:01:43.638039 29698 service_pool.cc:138] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:36260 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 939)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 959)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 828)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 833)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 831)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 846)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 842)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 841)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 848)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 850)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 843)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 838)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 849)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 845)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 847)
W20240501 20:01:43.633137 31135 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.635s user 0.012s sys 0.011s
I20240501 20:01:43.621030 31019 raft_consensus.cc:482] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:43.652376 31042 raft_consensus.cc:3032] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:01:43.661159 31042 raft_consensus.cc:729] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:43.662122 31042 consensus_queue.cc:260] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } }
W20240501 20:01:43.663201 29543 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.174s user 0.006s sys 0.082s
W20240501 20:01:43.663602 29543 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.302s user 0.006s sys 0.082s
W20240501 20:01:43.663857 29543 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 1.36033e+06us
I20240501 20:01:43.665887 29802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9472c53f79ec423f82c396bd980674e2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:43.655779 31292 raft_consensus.cc:2726] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861) with higher term. Message: Invalid argument: T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 888d99522e4844118f8f1cf26f268c8c for earlier term 4. Current term is 5.
I20240501 20:01:43.667634 31153 leader_election.cc:288] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:43.660001 31019 raft_consensus.cc:504] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:43.671583 31019 leader_election.cc:288] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:43.652482 31258 raft_consensus.cc:3032] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:01:43.653940 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c597f15e844f30837012baed4b0501" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:43.646823 29773 leader_election.cc:302] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
W20240501 20:01:43.688472 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 48156.6us
W20240501 20:01:43.689266 29773 leader_election.cc:334] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 1.862s (SENT)
W20240501 20:01:43.654726 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 434393us
I20240501 20:01:43.688144 29698 service_pool.cc:138] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:36260 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 939)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 959)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 828)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 833)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 831)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 846)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 842)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 841)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 848)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 850)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 843)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 838)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 849)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 845)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 847)
W20240501 20:01:43.693958 29624 leader_election.cc:334] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 1.543s (SENT)
I20240501 20:01:43.694499 29624 leader_election.cc:302] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
W20240501 20:01:43.695636 29624 leader_election.cc:334] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 1.596s (SENT)
I20240501 20:01:43.631338 31020 raft_consensus.cc:482] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 888d99522e4844118f8f1cf26f268c8c)
I20240501 20:01:43.636408 31206 raft_consensus.cc:2726] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 20:01:43.622766 30849 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.040s user 0.014s sys 0.009s
I20240501 20:01:43.651654 29807 raft_consensus.cc:1226] T 1ca3948261374cef8b1b206a43c00173 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 1. Current term is 2. Ops: []
W20240501 20:01:43.646600 29656 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.936s user 0.003s sys 0.034s
W20240501 20:01:43.728700 29656 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.018s user 0.003s sys 0.034s
I20240501 20:01:43.708585 29624 leader_election.cc:302] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:43.720250 29801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41858 (request call id 970) took 1411 ms. Trace:
W20240501 20:01:43.721750 29773 connection.cc:232] RPC call timeout handler was delayed by -1.1783371670001088s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2191s for a response
I20240501 20:01:43.729419 29801 rpcz_store.cc:276] 0501 20:01:42.309298 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:42.344076 (+ 34778us) service_pool.cc:225] Handling call
0501 20:01:42.344369 (+ 293us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:42.344593 (+ 224us) raft_consensus.cc:1520] Early marking committed up to index 2
0501 20:01:43.255791 (+911198us) log.cc:842] Serialized 14 byte log entry
0501 20:01:43.719873 (+464082us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:43.719931 (+ 58us) raft_consensus.cc:1631] Marking committed up to 2
0501 20:01:43.719972 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:43.720037 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:43.720196 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2009,"prepare.queue_time_us":16197,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":30,"raft.queue_time_us":25761,"thread_start_us":1326978,"threads_started":2}
W20240501 20:01:43.730281 29773 connection.cc:232] RPC call timeout handler was delayed by -1.1649124590001065s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17270000000000002s for a response
W20240501 20:01:43.730423 29701 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.611s user 0.004s sys 0.017s
W20240501 20:01:43.730708 29773 connection.cc:232] RPC call timeout handler was delayed by -1.1558588220000274s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16910000000000003s for a response
I20240501 20:01:43.655606 31132 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:40670 (request call id 46) took 5932 ms. Trace:
W20240501 20:01:43.731698 29773 connection.cc:232] RPC call timeout handler was delayed by -1.1157191679999414s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1564s for a response
W20240501 20:01:43.736804 29773 connection.cc:232] RPC call timeout handler was delayed by -1.0398259959999905s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2033s for a response
I20240501 20:01:43.687026 31258 raft_consensus.cc:729] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:43.723014 31019 raft_consensus.cc:2726] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20240501 20:01:43.732524 29624 leader_election.cc:334] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 1.629s (SENT)
I20240501 20:01:43.721992 29698 service_pool.cc:138] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:36260 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 939)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 959)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 828)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 833)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 831)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 846)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 842)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 841)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 848)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 850)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 843)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 838)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 849)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 845)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 847)
I20240501 20:01:43.696586 31020 raft_consensus.cc:504] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:43.751770 29801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e89c0572e4ee7be8e5315b459bdb3" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:43.696930 31170 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 3.242s user 0.014s sys 0.024s
I20240501 20:01:43.753808 29624 leader_election.cc:302] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
W20240501 20:01:43.731027 29701 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.612s user 0.004s sys 0.018s
I20240501 20:01:43.756884 29801 raft_consensus.cc:2454] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 6.
W20240501 20:01:43.731144 30384 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 437570us
I20240501 20:01:43.732184 31267 raft_consensus.cc:2726] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240501 20:01:43.763991 29624 leader_election.cc:334] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 1.801s (SENT)
I20240501 20:01:43.767405 29624 leader_election.cc:302] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:43.767042 29801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77be7101c0fe4085b6706296f609870a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:43.758766 31170 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 3.303s user 0.014s sys 0.033s
I20240501 20:01:43.734680 31257 raft_consensus.cc:2726] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:43.769982 29801 raft_consensus.cc:2454] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 2.
W20240501 20:01:43.737167 29773 connection.cc:232] RPC call timeout handler was delayed by -1.0326335830002336s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.227s for a response
W20240501 20:01:43.764029 29700 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.615s user 0.011s sys 0.020s
I20240501 20:01:43.736943 31292 consensus_queue.cc:1035] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.029s
W20240501 20:01:43.777247 29700 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.648s user 0.012s sys 0.020s
W20240501 20:01:43.773188 29624 leader_election.cc:334] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 2.080s (SENT)
I20240501 20:01:43.777015 29698 service_pool.cc:138] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:36260 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 939)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 959)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 828)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 833)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 831)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 846)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 842)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 841)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 848)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 850)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 843)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 838)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 849)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 845)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 847)
I20240501 20:01:43.766973 31210 raft_consensus.cc:482] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
W20240501 20:01:43.765477 31234 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.597s user 0.011s sys 0.026s
W20240501 20:01:43.818331 29773 leader_election.cc:334] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 1.727s (SENT)
I20240501 20:01:43.786931 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48fd0a8e289401aa7417c42ff2c81b2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:43.802587 29804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41858 (request call id 971) took 1482 ms. Trace:
I20240501 20:01:43.824880 29801 raft_consensus.cc:1226] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20240501 20:01:43.783174 29624 leader_election.cc:302] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:43.770684 31233 raft_consensus.cc:482] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:43.797689 31170 raft_consensus.cc:482] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:43.816712 31235 raft_consensus.cc:482] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240501 20:01:43.844066 29735 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 797) took 3143 ms (client timeout 2226 ms). Trace:
W20240501 20:01:43.877035 29735 rpcz_store.cc:269] 0501 20:01:40.700836 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:43.843833 (+3142997us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:43.843978 (+ 145us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240501 20:01:43.775342 31267 raft_consensus.cc:2726] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:43.825975 31257 raft_consensus.cc:482] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:43.882086 31257 raft_consensus.cc:504] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } }
I20240501 20:01:43.884404 31324 consensus_queue.cc:1035] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20240501 20:01:43.885182 31257 leader_election.cc:288] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:43.885717 31020 leader_election.cc:288] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:43.886379 31204 consensus_queue.cc:1035] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:43.859810 31153 raft_consensus.cc:2726] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:01:43.889578 30415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd5e83faaff4e2da4e4cc22ba410d33" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c" is_pre_election: true
I20240501 20:01:43.889901 30414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e89c0572e4ee7be8e5315b459bdb3" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c" is_pre_election: true
I20240501 20:01:43.891973 30417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e07220db7444ae97f294d70eda6a93" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c" is_pre_election: true
W20240501 20:01:43.895504 29733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 786) took 3199 ms (client timeout 2186 ms). Trace:
I20240501 20:01:43.896764 31197 raft_consensus.cc:504] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:43.898579 29652 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59136 (request call id 775) took 3465 ms (client timeout 1944 ms). Trace:
W20240501 20:01:43.898947 29652 rpcz_store.cc:269] 0501 20:01:40.433116 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:40.434679 (+ 1563us) service_pool.cc:225] Handling call
0501 20:01:43.898413 (+3463734us) spinlock_profiling.cc:243] Waited 1.73 s on lock 0x7b5400394ed4. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787f4fe 00007fe45c7bd9d7 00007fe455ea1c14 00007fe455ea1ba2 00007fe455ea1ad1 00007fe455ea1a4c 00007fe455ea0d02 00007fe455ca2bcc 00007fe455ca2006 00007fe455ca19af
0501 20:01:43.898509 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3973510400}
W20240501 20:01:43.895857 29733 rpcz_store.cc:269] 0501 20:01:40.696181 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:43.895351 (+3199170us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:43.895451 (+ 100us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240501 20:01:43.900192 31245 raft_consensus.cc:3032] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Stepping down as leader of term 4
I20240501 20:01:43.900588 31245 raft_consensus.cc:729] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
W20240501 20:01:43.900713 29733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912) took 2889 ms (client timeout 1925 ms). Trace:
W20240501 20:01:43.901057 29733 rpcz_store.cc:269] 0501 20:01:41.011272 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:43.900563 (+2889291us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:43.900663 (+ 100us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240501 20:01:43.901787 29733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907) took 2948 ms (client timeout 2018 ms). Trace:
I20240501 20:01:43.901356 31245 consensus_queue.cc:260] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:43.902139 29733 rpcz_store.cc:269] 0501 20:01:40.952997 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:43.901629 (+2948632us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:43.901722 (+ 93us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240501 20:01:43.902423 31245 raft_consensus.cc:3037] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Advancing to term 5
W20240501 20:01:43.902860 29733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916) took 2747 ms (client timeout 1826 ms). Trace:
W20240501 20:01:43.903206 29733 rpcz_store.cc:269] 0501 20:01:41.155017 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:43.902723 (+2747706us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:43.902815 (+ 92us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240501 20:01:43.893905 29731 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 889) took 3267 ms (client timeout 1722 ms). Trace:
W20240501 20:01:43.904753 29731 rpcz_store.cc:269] 0501 20:01:40.626298 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:40.626555 (+ 257us) service_pool.cc:225] Handling call
0501 20:01:43.893690 (+3267135us) spinlock_profiling.cc:243] Waited 1.8 s on lock 0x7b54001063d4. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787f4fe 00007fe45c7bd9d7 00007fe455ea1c14 00007fe455ea1ba2 00007fe455ea1ad1 00007fe455ea1a4c 00007fe455ea0d02 00007fe455ca2bcc 00007fe455ca2006 00007fe455ca19af
0501 20:01:43.893820 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4145939712}
W20240501 20:01:43.905653 29731 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903) took 2978 ms (client timeout 2111 ms). Trace:
W20240501 20:01:43.905956 29731 rpcz_store.cc:269] 0501 20:01:40.926968 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:43.905504 (+2978536us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:43.905602 (+ 98us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240501 20:01:43.906687 29731 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915) took 2788 ms (client timeout 1921 ms). Trace:
W20240501 20:01:43.906896 29731 rpcz_store.cc:269] 0501 20:01:41.118606 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:43.906553 (+2787947us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:43.906652 (+ 99us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240501 20:01:43.906155 29728 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 812) took 3201 ms (client timeout 2080 ms). Trace:
W20240501 20:01:43.909101 29728 rpcz_store.cc:269] 0501 20:01:40.704929 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:43.905986 (+3201057us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:43.906100 (+ 114us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240501 20:01:43.909992 29728 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818) took 3098 ms (client timeout 2270 ms). Trace:
W20240501 20:01:43.910348 29728 rpcz_store.cc:269] 0501 20:01:40.811349 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:43.909812 (+3098463us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:43.909937 (+ 125us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240501 20:01:43.911108 29728 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901) took 3036 ms (client timeout 2231 ms). Trace:
I20240501 20:01:43.880182 31235 raft_consensus.cc:504] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:43.911432 29728 rpcz_store.cc:269] 0501 20:01:40.874496 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:43.910961 (+3036465us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:43.911055 (+ 94us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240501 20:01:43.912035 31291 log.cc:924] Time spent T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 0.622s user 0.000s sys 0.002s
W20240501 20:01:43.860145 29727 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 801) took 3158 ms (client timeout 2010 ms). Trace:
W20240501 20:01:43.912678 29728 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792) took 3214 ms (client timeout 2422 ms). Trace:
W20240501 20:01:43.912468 29727 rpcz_store.cc:269] 0501 20:01:40.702106 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:41.833406 (+1131300us) service_pool.cc:225] Handling call
0501 20:01:43.859949 (+2026543us) spinlock_profiling.cc:243] Waited 166 ms on lock 0x7b5400141254. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787f4fe 00007fe45c7bd9d7 00007fe455ea1c14 00007fe455ea1ba2 00007fe455ea1ad1 00007fe455ea1a4c 00007fe455ea0d02 00007fe455ca2bcc 00007fe455ca2006 00007fe455ca19af
0501 20:01:43.860085 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":381656960}
I20240501 20:01:43.913239 31235 leader_election.cc:288] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:43.913333 29728 rpcz_store.cc:269] 0501 20:01:40.697763 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:43.912536 (+3214773us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:43.912633 (+ 97us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240501 20:01:43.913710 29727 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913) took 2858 ms (client timeout 2068 ms). Trace:
W20240501 20:01:43.914278 29727 rpcz_store.cc:269] 0501 20:01:41.055522 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:43.913542 (+2858020us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:43.913658 (+ 116us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240501 20:01:43.914723 29728 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918) took 2665 ms (client timeout 1892 ms). Trace:
I20240501 20:01:43.914983 31282 raft_consensus.cc:3032] T 1ca3948261374cef8b1b206a43c00173 P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Stepping down as leader of term 1
W20240501 20:01:43.915167 29727 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917) took 2717 ms (client timeout 2080 ms). Trace:
I20240501 20:01:43.915448 31282 raft_consensus.cc:729] T 1ca3948261374cef8b1b206a43c00173 P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
W20240501 20:01:43.903882 29733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816) took 3097 ms (client timeout 2191 ms). Trace:
W20240501 20:01:43.915774 31320 log.cc:924] Time spent T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d: Append to log took a long time: real 0.098s user 0.000s sys 0.000s
W20240501 20:01:43.916191 29733 rpcz_store.cc:269] 0501 20:01:40.806326 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:43.903742 (+3097416us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:43.903834 (+ 92us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240501 20:01:43.916366 31282 consensus_queue.cc:260] T 1ca3948261374cef8b1b206a43c00173 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:43.917184 29733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924) took 2212 ms (client timeout 1596 ms). Trace:
I20240501 20:01:43.917480 31282 raft_consensus.cc:3037] T 1ca3948261374cef8b1b206a43c00173 P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:01:43.915210 29728 rpcz_store.cc:269] 0501 20:01:41.249100 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:43.914527 (+2665427us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:43.914653 (+ 126us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240501 20:01:43.921223 29728 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923) took 2240 ms (client timeout 1622 ms). Trace:
I20240501 20:01:43.921340 29726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 874) took 3351 ms. Trace:
W20240501 20:01:43.921550 29728 rpcz_store.cc:269] 0501 20:01:41.681232 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:43.921074 (+2239842us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:43.921176 (+ 102us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240501 20:01:43.922433 29728 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787) took 3226 ms (client timeout 2654 ms). Trace:
W20240501 20:01:43.922886 29728 rpcz_store.cc:269] 0501 20:01:40.696455 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:43.922296 (+3225841us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:43.922389 (+ 93us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240501 20:01:43.921705 29726 rpcz_store.cc:276] 0501 20:01:40.570180 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:40.570271 (+ 91us) service_pool.cc:225] Handling call
0501 20:01:43.188165 (+2617894us) spinlock_profiling.cc:243] Waited 4.3 ms on lock 0x7b58001b3098. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe454797b51 00007fe4549193b1 00007fe454919262 00007fe45d80b889 00007fe45c7483c0 00007fe45c80e3bb 00007fe45c80e441 00007fe45c7d2060 00007fe45c7bd5a9 00007fe45c7bd052 00007fe455e9f694
0501 20:01:43.188391 (+ 226us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:43.292622 (+104231us) raft_consensus.cc:1520] Early marking committed up to index 3
0501 20:01:43.723635 (+431013us) log.cc:842] Serialized 14 byte log entry
0501 20:01:43.869936 (+146301us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:43.869979 (+ 43us) raft_consensus.cc:1631] Marking committed up to 3
0501 20:01:43.870022 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:43.921115 (+ 51093us) spinlock_profiling.cc:243] Waited 104 ms on lock 0x7b54001eebd4. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787c1a1 00007fe45787abee 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc 00007fe455e9e782 00007fe455ca2bcc 00007fe455ca2006
0501 20:01:43.921158 (+ 43us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:43.921292 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":514392,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"raft.queue_time_us":138,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":2709,"spinlock_wait_cycles":248852224,"thread_start_us":539258,"threads_started":2}
I20240501 20:01:43.831897 31079 consensus_queue.cc:1046] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.095s
I20240501 20:01:43.924264 29699 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 962) took 1301 ms. Trace:
I20240501 20:01:43.924820 31079 consensus_queue.cc:1225] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer ee0d97247d394a8facfa774f15f3a67d log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20240501 20:01:43.924876 29699 rpcz_store.cc:276] 0501 20:01:42.622547 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:43.924205 (+1301658us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240501 20:01:43.831648 31232 consensus_queue.cc:1046] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.094s
I20240501 20:01:43.837261 31229 consensus_queue.cc:1046] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.089s
I20240501 20:01:43.925776 29699 service_pool.cc:138] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 939)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 959)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 828)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 833)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 831)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 846)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 842)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 841)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 848)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 850)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 843)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 838)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 849)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 845)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 847)
I20240501 20:01:43.863906 31181 consensus_queue.cc:1046] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 3, Time since last communication: 0.001s
W20240501 20:01:43.833753 29773 leader_election.cc:334] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 1.770s (SENT)
I20240501 20:01:43.837818 29802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5525fc4b82a44b0991c3dd6098fde81" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:43.928012 29773 leader_election.cc:302] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:43.928206 29802 raft_consensus.cc:2454] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 4.
I20240501 20:01:43.869508 29808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41864 (request call id 1051) took 1458 ms. Trace:
W20240501 20:01:43.928884 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 95173.2us
I20240501 20:01:43.928874 29808 rpcz_store.cc:276] 0501 20:01:42.410893 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:43.120020 (+709127us) service_pool.cc:225] Handling call
0501 20:01:43.120306 (+ 286us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:43.635762 (+515456us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:43.869318 (+233556us) spinlock_profiling.cc:243] Waited 514 ms on lock 0x7b54001f4354. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787c1a1 00007fe45787abee 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc 00007fe455e9e782 00007fe455ca2bcc 00007fe455ca2006
0501 20:01:43.869458 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1182347776}
W20240501 20:01:43.861177 31308 log.cc:924] Time spent T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d: Append to log took a long time: real 0.105s user 0.000s sys 0.000s
I20240501 20:01:43.929520 29653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59142 (request call id 932) took 1564 ms. Trace:
I20240501 20:01:43.772890 31311 raft_consensus.cc:2726] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240501 20:01:43.847311 31234 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.679s user 0.012s sys 0.026s
W20240501 20:01:43.907789 29731 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905) took 2961 ms (client timeout 2104 ms). Trace:
W20240501 20:01:43.939405 29731 rpcz_store.cc:269] 0501 20:01:40.945680 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:43.907299 (+2961619us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:43.907357 (+ 58us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240501 20:01:43.927204 31229 raft_consensus.cc:3032] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Stepping down as leader of term 4
W20240501 20:01:43.915808 29727 rpcz_store.cc:269] 0501 20:01:41.198138 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:43.915012 (+2716874us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:43.915119 (+ 107us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240501 20:01:43.840916 31220 leader_election.cc:288] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:43.929926 29653 rpcz_store.cc:276] 0501 20:01:42.364901 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:42.365037 (+ 136us) service_pool.cc:225] Handling call
0501 20:01:42.454979 (+ 89942us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:42.455523 (+ 544us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:43.929330 (+1473807us) spinlock_profiling.cc:243] Waited 83.9 ms on lock 0x7b54001c3250. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787ac59 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc 00007fe455e9e782 00007fe455ca2bcc 00007fe455ca2006 00007fe455ca19af
0501 20:01:43.929464 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":192938752}
I20240501 20:01:43.931123 29656 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59136 (request call id 827) took 1363 ms. Trace:
W20240501 20:01:43.932395 29773 leader_election.cc:334] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 1.843s (SENT)
I20240501 20:01:43.927989 31181 consensus_queue.cc:1225] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Peer ee0d97247d394a8facfa774f15f3a67d log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240501 20:01:43.829479 31210 raft_consensus.cc:504] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:43.948493 29734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 876) took 3374 ms. Trace:
W20240501 20:01:43.949043 29803 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41858 (request call id 940) took 3506 ms (client timeout 2111 ms). Trace:
W20240501 20:01:44.006532 29803 rpcz_store.cc:269] 0501 20:01:40.442926 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:41.019535 (+576609us) service_pool.cc:225] Handling call
0501 20:01:43.948871 (+2929336us) spinlock_profiling.cc:243] Waited 532 ms on lock 0x7b54001f4d54. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787f4fe 00007fe45c7bd9d7 00007fe455ea1c14 00007fe455ea1ba2 00007fe455ea1ad1 00007fe455ea1a4c 00007fe455ea0d02 00007fe455ca2bcc 00007fe455ca2006 00007fe455ca19af
0501 20:01:43.948988 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1222998912}
I20240501 20:01:43.949568 29805 raft_consensus.cc:1226] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 4. Current term is 5. Ops: []
I20240501 20:01:43.858044 29804 rpcz_store.cc:276] 0501 20:01:42.320101 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:42.549663 (+229562us) service_pool.cc:225] Handling call
0501 20:01:42.549999 (+ 336us) raft_consensus.cc:1457] Updating replica for 1 ops
0501 20:01:42.550385 (+ 386us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:42.550408 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0501 20:01:42.550855 (+ 447us) log.cc:842] Serialized 26 byte log entry
0501 20:01:43.558847 (+1007992us) raft_consensus.cc:1631] Marking committed up to 1
0501 20:01:43.558897 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:43.558994 (+ 97us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0501 20:01:43.802360 (+243366us) raft_consensus.cc:1671] finished
0501 20:01:43.802383 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:43.802525 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":201,"raft.run_cpu_time_us":59,"raft.run_wall_time_us":58,"thread_start_us":1007607,"threads_started":1,"wal-append.queue_time_us":1221083}
W20240501 20:01:43.917490 29733 rpcz_store.cc:269] 0501 20:01:41.704940 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:43.917035 (+2212095us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:43.917140 (+ 105us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240501 20:01:44.009261 29733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931) took 2145 ms (client timeout 1629 ms). Trace:
W20240501 20:01:44.009828 29733 rpcz_store.cc:269] 0501 20:01:41.863907 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:44.009097 (+2145190us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:44.009204 (+ 107us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240501 20:01:43.953104 29731 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929) took 2121 ms (client timeout 1543 ms). Trace:
W20240501 20:01:43.952877 29726 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909) took 2960 ms (client timeout 2397 ms). Trace:
W20240501 20:01:43.949647 31246 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.614s user 0.002s sys 0.006s
I20240501 20:01:43.876456 31170 raft_consensus.cc:504] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:43.734701 31132 rpcz_store.cc:276] 0501 20:01:37.720288 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:37.721571 (+ 1283us) service_pool.cc:225] Handling call
0501 20:01:38.108477 (+386906us) ts_tablet_manager.cc:903] T be383be783234855878cb8fd442c7b05 P 888d99522e4844118f8f1cf26f268c8c: Initiating tablet copy from peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
0501 20:01:42.008609 (+3900132us) tablet_copy_client.cc:353] Tablet copy session begun
0501 20:01:42.389167 (+380558us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:42.707259 (+318092us) tablet_metadata.cc:641] Metadata flushed
0501 20:01:42.922530 (+215271us) tablet_copy_client.cc:448] Replaced tombstoned tablet metadata.
0501 20:01:43.532188 (+609658us) tablet_copy_client.cc:403] Wrote new consensus metadata
0501 20:01:43.579299 (+ 47111us) tablet_replica.cc:205] Creating consensus instance
0501 20:01:43.640079 (+ 60780us) ts_tablet_manager.cc:958] Replica registered.
0501 20:01:43.646271 (+ 6192us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":37568,"tablet-copy.queue_time_us":259458,"thread_start_us":2086513,"threads_started":5}
I20240501 20:01:43.959671 29772 leader_election.cc:302] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:43.959003 29656 rpcz_store.cc:276] 0501 20:01:42.567746 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:42.584766 (+ 17020us) service_pool.cc:225] Handling call
0501 20:01:42.585105 (+ 339us) raft_consensus.cc:1457] Updating replica for 1 ops
0501 20:01:42.607550 (+ 22445us) raft_consensus.cc:1520] Early marking committed up to index 2
0501 20:01:43.729057 (+1121507us) log.cc:842] Serialized 14 byte log entry
0501 20:01:43.835742 (+106685us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0501 20:01:43.874024 (+ 38282us) log.cc:842] Serialized 26 byte log entry
0501 20:01:43.874217 (+ 193us) raft_consensus.cc:1631] Marking committed up to 2
0501 20:01:43.874266 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:43.874354 (+ 88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0501 20:01:43.930940 (+ 56586us) raft_consensus.cc:1671] finished
0501 20:01:43.930957 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:43.931065 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":103945,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":22318,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":1442,"thread_start_us":1101255,"threads_started":2,"wal-append.queue_time_us":130488}
I20240501 20:01:43.929533 31060 raft_consensus.cc:2726] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240501 20:01:44.205638 29730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 790) took 3508 ms (client timeout 2034 ms). Trace:
W20240501 20:01:44.206168 29730 rpcz_store.cc:269] 0501 20:01:40.697241 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:44.205415 (+3508174us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:44.205564 (+ 149us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240501 20:01:43.859930 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 86760.4us
W20240501 20:01:44.208294 29624 leader_election.cc:334] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 2.309s (SENT)
I20240501 20:01:44.208803 29624 leader_election.cc:302] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:44.005018 29734 rpcz_store.cc:276] 0501 20:01:40.573800 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:40.573903 (+ 103us) service_pool.cc:225] Handling call
0501 20:01:40.574103 (+ 200us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:40.576873 (+ 2770us) raft_consensus.cc:1520] Early marking committed up to index 2
0501 20:01:41.186814 (+609941us) log.cc:842] Serialized 14 byte log entry
0501 20:01:41.478926 (+292112us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:41.478973 (+ 47us) raft_consensus.cc:1631] Marking committed up to 2
0501 20:01:41.479006 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:43.948277 (+2469271us) spinlock_profiling.cc:243] Waited 2.61 ms on lock 0x7b5400290254. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787c1a1 00007fe45787abee 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc 00007fe455e9e782 00007fe455ca2bcc 00007fe455ca2006
0501 20:01:43.948323 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:43.948443 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8841,"prepare.queue_time_us":122867,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":894,"raft.queue_time_us":3758,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":11532,"spinlock_wait_cycles":5997824,"thread_start_us":876481,"threads_started":2,"wal-append.queue_time_us":333594,"wal-append.run_cpu_time_us":2264,"wal-append.run_wall_time_us":1186334}
I20240501 20:01:43.874444 29698 service_pool.cc:138] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:36260 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 912)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 907)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 916)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 816)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 903)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 915)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 905)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 818)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 901)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 913)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 918)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 917)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 923)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 787)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 909)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 939)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 959)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 828)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 833)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 831)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 846)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 842)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 841)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 848)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 850)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 843)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 838)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 849)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 845)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 847)
W20240501 20:01:43.952824 29728 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 925) took 2198 ms (client timeout 1629 ms). Trace:
W20240501 20:01:44.010581 29731 rpcz_store.cc:269] 0501 20:01:41.828045 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:43.948906 (+2120861us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:43.948999 (+ 93us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240501 20:01:44.215065 29731 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 922) took 2558 ms (client timeout 2084 ms). Trace:
W20240501 20:01:44.102829 29733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 926) took 2345 ms (client timeout 1801 ms). Trace:
W20240501 20:01:44.216233 29733 rpcz_store.cc:269] 0501 20:01:41.757795 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:44.011517 (+2253722us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:44.102754 (+ 91237us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240501 20:01:44.217221 29733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 928) took 2417 ms (client timeout 2035 ms). Trace:
W20240501 20:01:44.217592 29733 rpcz_store.cc:269] 0501 20:01:41.799814 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:44.217023 (+2417209us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:44.217155 (+ 132us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240501 20:01:44.214509 29734 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 919) took 2881 ms (client timeout 2309 ms). Trace:
I20240501 20:01:43.775174 31258 consensus_queue.cc:260] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:44.149811 29772 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 190193us
W20240501 20:01:44.207065 29730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 900) took 3359 ms (client timeout 2784 ms). Trace:
W20240501 20:01:44.237722 29730 rpcz_store.cc:269] 0501 20:01:40.847392 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:44.206890 (+3359498us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:44.206999 (+ 109us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240501 20:01:43.949715 31229 raft_consensus.cc:729] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:43.875838 31233 raft_consensus.cc:504] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:44.145373 29732 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 906) took 3196 ms (client timeout 1543 ms). Trace:
I20240501 20:01:44.145946 29657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59136 (request call id 821) took 1832 ms. Trace:
W20240501 20:01:44.218482 29733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 939) took 2185 ms (client timeout 1830 ms). Trace:
W20240501 20:01:44.010738 29726 rpcz_store.cc:269] 0501 20:01:40.963691 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:43.923541 (+2959850us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:43.923624 (+ 83us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240501 20:01:44.214532 29728 rpcz_store.cc:269] 0501 20:01:41.724658 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:43.923419 (+2198761us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:43.923526 (+ 107us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240501 20:01:44.215559 29731 rpcz_store.cc:269] 0501 20:01:41.656887 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:44.214859 (+2557972us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:44.214991 (+ 132us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240501 20:01:44.132117 29727 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 921) took 2490 ms (client timeout 1780 ms). Trace:
W20240501 20:01:44.245435 29731 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 930) took 2401 ms (client timeout 2124 ms). Trace:
W20240501 20:01:44.249289 29731 rpcz_store.cc:269] 0501 20:01:41.844322 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:44.245258 (+2400936us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:44.245366 (+ 108us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240501 20:01:44.249981 29731 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 927) took 2458 ms (client timeout 2246 ms). Trace:
W20240501 20:01:44.250298 29731 rpcz_store.cc:269] 0501 20:01:41.792012 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:44.249844 (+2457832us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:44.249932 (+ 88us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240501 20:01:44.250960 29624 leader_election.cc:334] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 1.830s (SENT)
I20240501 20:01:44.251413 29624 leader_election.cc:302] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:44.241959 29772 leader_election.cc:302] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
W20240501 20:01:44.219043 29734 rpcz_store.cc:269] 0501 20:01:41.332967 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:44.214308 (+2881341us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:44.214440 (+ 132us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240501 20:01:44.244067 29732 rpcz_store.cc:269] 0501 20:01:40.949327 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:41.022184 (+ 72857us) service_pool.cc:225] Handling call
0501 20:01:44.145226 (+3123042us) spinlock_profiling.cc:243] Waited 763 ms on lock 0x7b54002d7dd4. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787f4fe 00007fe45c7bd9d7 00007fe455ea1c14 00007fe455ea1ba2 00007fe455ea1ad1 00007fe455ea1a4c 00007fe455ea0d02 00007fe455ca2bcc 00007fe455ca2006 00007fe455ca19af
0501 20:01:44.145322 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1755359104}
W20240501 20:01:44.246567 29730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 959) took 1857 ms (client timeout 1521 ms). Trace:
W20240501 20:01:44.256994 29730 rpcz_store.cc:269] 0501 20:01:42.389070 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:44.246415 (+1857345us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:44.246504 (+ 89us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240501 20:01:44.257200 29732 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 896) took 3451 ms (client timeout 3257 ms). Trace:
W20240501 20:01:44.257714 29730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 932) took 2377 ms (client timeout 2231 ms). Trace:
W20240501 20:01:44.258036 29730 rpcz_store.cc:269] 0501 20:01:41.880763 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:44.257583 (+2376820us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:44.257668 (+ 85us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240501 20:01:44.258652 29730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 60) took 2305 ms (client timeout 2176 ms). Trace:
W20240501 20:01:44.258921 29730 rpcz_store.cc:269] 0501 20:01:41.953095 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:44.258509 (+2305414us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:44.258604 (+ 95us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240501 20:01:44.244244 29657 rpcz_store.cc:276] 0501 20:01:42.313861 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:42.328877 (+ 15016us) service_pool.cc:225] Handling call
0501 20:01:42.329106 (+ 229us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:42.479411 (+150305us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:44.145734 (+1666323us) spinlock_profiling.cc:243] Waited 150 ms on lock 0x7b540034e4d4. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787c1a1 00007fe45787abee 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc 00007fe455e9e782 00007fe455ca2bcc 00007fe455ca2006
0501 20:01:44.145894 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":344210176}
W20240501 20:01:44.244786 29726 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 933) took 2273 ms (client timeout 1948 ms). Trace:
W20240501 20:01:44.244977 29728 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 58) took 2324 ms (client timeout 2020 ms). Trace:
W20240501 20:01:44.260169 29728 rpcz_store.cc:269] 0501 20:01:41.920384 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:44.244800 (+2324416us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:44.244912 (+ 112us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240501 20:01:44.245045 31132 tablet_copy_client.cc:783] T be383be783234855878cb8fd442c7b05 P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Starting download of 0 data blocks...
W20240501 20:01:44.213094 31246 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.877s user 0.003s sys 0.010s
W20240501 20:01:44.263013 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 389786us
I20240501 20:01:44.263362 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b337b8e3e38d4ad6b1076fbd0d0a1609" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
W20240501 20:01:44.264168 29728 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 833) took 1445 ms (client timeout 1564 ms). Trace:
W20240501 20:01:44.264501 29728 rpcz_store.cc:269] 0501 20:01:42.818290 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:44.263123 (+1444833us) service_pool.cc:225] Handling call
0501 20:01:44.264117 (+ 994us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:44.265301 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e9f75387564275a60d38e9a910319c" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:44.265749 29728 raft_consensus.cc:2454] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 2.
W20240501 20:01:44.266075 29728 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 831) took 1448 ms (client timeout 1886 ms). Trace:
W20240501 20:01:44.244391 29733 rpcz_store.cc:269] 0501 20:01:42.033121 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:44.218309 (+2185188us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:44.218426 (+ 117us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240501 20:01:44.259893 29726 rpcz_store.cc:269] 0501 20:01:41.971209 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:44.244628 (+2273419us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:44.244741 (+ 113us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240501 20:01:44.266317 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0374bb790ed345ed90452595f71383a2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
W20240501 20:01:44.257553 29732 rpcz_store.cc:269] 0501 20:01:40.805642 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:44.257062 (+3451420us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:44.257152 (+ 90us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240501 20:01:44.239986 30384 leader_election.cc:334] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 2.020s (SENT)
I20240501 20:01:44.149647 31140 raft_consensus.cc:482] T 3c5345f6eb62449ba69b4a9c2e349b31 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240501 20:01:44.268991 29730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 828) took 1758 ms (client timeout 1691 ms). Trace:
W20240501 20:01:44.269407 29730 rpcz_store.cc:269] 0501 20:01:42.510384 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:44.268843 (+1758459us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:44.268940 (+ 97us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240501 20:01:44.266247 29728 rpcz_store.cc:269] 0501 20:01:42.817691 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:44.265131 (+1447440us) service_pool.cc:225] Handling call
0501 20:01:44.266040 (+ 909us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:44.267832 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfac893b4b4f457694a4eb184b6abb53" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
W20240501 20:01:44.250907 29731 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 908) took 3291 ms (client timeout 3083 ms). Trace:
W20240501 20:01:44.271131 29731 rpcz_store.cc:269] 0501 20:01:40.959367 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:44.250784 (+3291417us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:44.250864 (+ 80us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240501 20:01:44.272195 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d86f8848a184a87947fd927c44e5a35" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:44.272522 29734 raft_consensus.cc:2396] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20240501 20:01:44.239287 31063 raft_consensus.cc:2726] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240501 20:01:44.273500 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649b71f9f267437ca51c758dd9ca6987" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:44.275791 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e38a2e91a3d48d09a22abce723af833" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:44.276188 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69915dd5992641e3a24f7f45f1cd5c0c" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:44.279160 29734 raft_consensus.cc:3037] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:44.280160 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9472c53f79ec423f82c396bd980674e2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:44.279654 29805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41864 (request call id 1015) took 3634 ms. Trace:
I20240501 20:01:44.281026 29805 rpcz_store.cc:276] 0501 20:01:40.645432 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:41.595324 (+949892us) service_pool.cc:225] Handling call
0501 20:01:43.949387 (+2354063us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:44.007423 (+ 58036us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:44.279155 (+271732us) spinlock_profiling.cc:243] Waited 2.35 s on lock 0x7b54001f4d50. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787ac59 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc 00007fe455e9e782 00007fe455ca2bcc 00007fe455ca2006 00007fe455ca19af
0501 20:01:44.279417 (+ 262us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5411722496}
I20240501 20:01:44.241676 31307 raft_consensus.cc:2781] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:01:44.282688 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c597f15e844f30837012baed4b0501" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:44.283656 29728 raft_consensus.cc:2396] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240501 20:01:44.283715 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a35893b91247a99912eba787d787cf" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:44.271612 31060 raft_consensus.cc:2781] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:01:44.285375 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfac893b4b4f457694a4eb184b6abb53" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:44.290539 29732 raft_consensus.cc:2396] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20240501 20:01:44.285632 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:44.268690 31279 raft_consensus.cc:2726] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240501 20:01:44.286875 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38cbeb0dc0c14e3d9182101eea885625" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
W20240501 20:01:44.287039 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 354645us
I20240501 20:01:44.269220 31258 raft_consensus.cc:3037] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:44.273188 31132 tablet_copy_client.cc:647] T be383be783234855878cb8fd442c7b05 P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Starting download of 1 WAL segments...
I20240501 20:01:44.293041 31212 consensus_queue.cc:1046] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:44.303932 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5525f546426c47a38edf7f56ad04b7ab" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:44.277424 31197 leader_election.cc:288] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:44.307601 29734 raft_consensus.cc:2454] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 4.
W20240501 20:01:44.287070 31290 log.cc:924] Time spent T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.996s user 0.000s sys 0.002s
W20240501 20:01:44.311424 31325 log.cc:924] Time spent T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20240501 20:01:44.311969 30417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e89c0572e4ee7be8e5315b459bdb3" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c" is_pre_election: true
I20240501 20:01:44.313154 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e38a2e91a3d48d09a22abce723af833" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
W20240501 20:01:44.249161 29727 rpcz_store.cc:269] 0501 20:01:41.624407 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:44.008472 (+2384065us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:44.114274 (+105802us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240501 20:01:44.310801 31205 consensus_queue.cc:1046] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 20:01:44.317210 29734 raft_consensus.cc:1226] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 3. Current term is 4. Ops: []
W20240501 20:01:44.317776 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 66835.4us
W20240501 20:01:44.327270 29624 leader_election.cc:334] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 2.035s (SENT)
I20240501 20:01:44.327342 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0374bb790ed345ed90452595f71383a2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:44.328976 29731 raft_consensus.cc:2379] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 4: Already voted for candidate ee0d97247d394a8facfa774f15f3a67d in this term.
I20240501 20:01:44.317970 29733 raft_consensus.cc:2454] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 4.
I20240501 20:01:44.322650 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d88139f89c47ff84cad674147aae0f" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:44.323841 29773 leader_election.cc:334] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 2.129s (SENT)
I20240501 20:01:44.330133 29773 leader_election.cc:302] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:44.325476 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8787bc4d6b44649b9138d906730110a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
W20240501 20:01:44.284873 30384 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 72101.3us
W20240501 20:01:44.330430 29624 leader_election.cc:334] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 2.246s (SENT)
I20240501 20:01:44.273908 31140 raft_consensus.cc:504] T 3c5345f6eb62449ba69b4a9c2e349b31 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:44.337920 29624 leader_election.cc:302] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
W20240501 20:01:44.339450 29773 leader_election.cc:334] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 2.191s (SENT)
W20240501 20:01:44.340479 29773 leader_election.cc:334] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 2.270s (SENT)
I20240501 20:01:44.340880 29773 leader_election.cc:302] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:44.332315 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a35893b91247a99912eba787d787cf" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:44.330938 29728 raft_consensus.cc:2454] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 3.
I20240501 20:01:44.334225 31170 leader_election.cc:288] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:44.336254 29732 raft_consensus.cc:1226] T 1ca3948261374cef8b1b206a43c00173 P fdfeac4190e045f290e003b2298d988d [term 2 LEADER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:44.294497 31307 raft_consensus.cc:686] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d [term 5 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:44.350847 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:44.349604 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece8d4ed84c34b748f7b6272b31c90ac" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:44.354998 29657 raft_consensus.cc:2454] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 4.
I20240501 20:01:44.356397 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e89c0572e4ee7be8e5315b459bdb3" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:44.357333 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49da4246f5f6438abe0e7e8c9a523d80" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:44.357903 29804 raft_consensus.cc:2454] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 4.
I20240501 20:01:44.353910 29728 raft_consensus.cc:2454] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 4.
I20240501 20:01:44.360220 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649b71f9f267437ca51c758dd9ca6987" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:44.361400 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77be7101c0fe4085b6706296f609870a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:44.362146 29728 raft_consensus.cc:2454] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 3.
I20240501 20:01:44.364104 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e07220db7444ae97f294d70eda6a93" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:44.364998 29731 raft_consensus.cc:2454] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 3.
I20240501 20:01:44.366894 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add9d7579351483fb268f89a44e95d8a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:44.369133 29728 raft_consensus.cc:1226] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20240501 20:01:44.370481 29728 raft_consensus.cc:1226] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 3. Current term is 4. Ops: []
I20240501 20:01:44.377681 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77be7101c0fe4085b6706296f609870a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:44.350975 31245 raft_consensus.cc:3032] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Stepping down as leader of term 4
I20240501 20:01:44.378466 29656 raft_consensus.cc:2454] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 3.
I20240501 20:01:44.356853 31205 consensus_queue.cc:1225] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer ee0d97247d394a8facfa774f15f3a67d log is divergent from this leader: its last log entry 4.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20240501 20:01:44.355610 31221 raft_consensus.cc:2726] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:44.406322 29731 raft_consensus.cc:1261] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240501 20:01:44.388685 29804 raft_consensus.cc:1226] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [term 4 LEADER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20240501 20:01:44.372668 31170 raft_consensus.cc:3037] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:44.350397 31279 raft_consensus.cc:2726] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:44.354995 31212 raft_consensus.cc:482] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:44.471570 29652 raft_consensus.cc:1261] T 2d86f8848a184a87947fd927c44e5a35 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 20:01:44.291108 31229 consensus_queue.cc:260] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:44.476945 29624 leader_election.cc:334] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 2.018s (SENT)
I20240501 20:01:44.480679 29728 raft_consensus.cc:3032] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:01:44.487962 29624 leader_election.cc:302] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:44.488024 29728 raft_consensus.cc:729] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:44.346076 31246 consensus_queue.cc:1046] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.126s
I20240501 20:01:44.488210 29656 raft_consensus.cc:1261] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20240501 20:01:44.491837 29624 leader_election.cc:334] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 2.104s (SENT)
I20240501 20:01:44.421461 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:44.489288 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5525fc4b82a44b0991c3dd6098fde81" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:44.493156 29657 raft_consensus.cc:3037] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Advancing to term 5
I20240501 20:01:44.492975 29624 leader_election.cc:302] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:44.489110 29728 consensus_queue.cc:260] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:44.494269 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd5e83faaff4e2da4e4cc22ba410d33" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:44.494576 29728 raft_consensus.cc:3037] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Advancing to term 4
W20240501 20:01:44.495159 29624 leader_election.cc:334] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 2.111s (SENT)
I20240501 20:01:44.418352 31245 raft_consensus.cc:729] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
W20240501 20:01:44.495887 29624 leader_election.cc:334] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 2.231s (SENT)
I20240501 20:01:44.484798 31267 raft_consensus.cc:482] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:44.486039 29802 raft_consensus.cc:1226] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 4. Current term is 5. Ops: []
I20240501 20:01:44.494971 29731 raft_consensus.cc:2454] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 5.
W20240501 20:01:44.418648 30384 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 53844.6us
W20240501 20:01:44.496989 29625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 242942us
I20240501 20:01:44.506223 29700 leader_election.cc:302] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:44.483343 30414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add9d7579351483fb268f89a44e95d8a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c" is_pre_election: true
I20240501 20:01:44.576398 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece8d4ed84c34b748f7b6272b31c90ac" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:44.499794 31229 raft_consensus.cc:3037] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:44.500149 31333 raft_consensus.cc:2726] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240501 20:01:44.507262 31328 log.cc:924] Time spent T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 0.189s user 0.000s sys 0.001s
W20240501 20:01:44.502285 31288 log.cc:924] Time spent T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 1.419s user 0.002s sys 0.008s
I20240501 20:01:44.580014 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f261d2a05bf4c838532049601694b4a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:44.586758 29728 raft_consensus.cc:1261] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20240501 20:01:44.501523 31279 raft_consensus.cc:2726] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20240501 20:01:44.588567 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 82440.9us
I20240501 20:01:44.598273 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38cbeb0dc0c14e3d9182101eea885625" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:44.568764 29657 raft_consensus.cc:2454] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 5.
I20240501 20:01:44.490442 31212 raft_consensus.cc:504] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:44.636659 29805 raft_consensus.cc:1226] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d [term 4 LEADER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 3. Current term is 4. Ops: []
I20240501 20:01:44.645985 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f261d2a05bf4c838532049601694b4a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:44.592242 31327 log.cc:924] Time spent T 456035a1cb3f4f7da52d0fc1dd1c656c P ee0d97247d394a8facfa774f15f3a67d: Append to log took a long time: real 0.215s user 0.000s sys 0.000s
I20240501 20:01:44.590145 31060 raft_consensus.cc:686] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [term 5 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:44.580837 31311 raft_consensus.cc:482] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:44.656440 31140 leader_election.cc:288] T 3c5345f6eb62449ba69b4a9c2e349b31 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:44.660310 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a2bba62f434d1c8489b55c8fec2187" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:44.660183 31208 consensus_queue.cc:1046] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 20:01:44.661499 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "456035a1cb3f4f7da52d0fc1dd1c656c" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:44.592238 31330 raft_consensus.cc:2781] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240501 20:01:44.594457 31233 leader_election.cc:288] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:44.687772 29728 pending_rounds.cc:76] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d: Aborting all ops after (but not including) 2
I20240501 20:01:44.688179 29728 pending_rounds.cc:97] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20240501 20:01:44.688494 29728 raft_consensus.cc:2866] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20240501 20:01:44.662074 31190 log.cc:924] Time spent T 00000000000000000000000000000000 P 995802798c1e438daecd368538bebecd: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20240501 20:01:44.648283 31210 leader_election.cc:288] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:44.572371 31267 raft_consensus.cc:504] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:44.698951 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69915dd5992641e3a24f7f45f1cd5c0c" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:44.510371 31307 consensus_queue.cc:237] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:44.669976 29802 raft_consensus.cc:1226] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d [term 4 LEADER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 3. Current term is 4. Ops: []
W20240501 20:01:44.717439 29624 leader_election.cc:334] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Remote error: Service unavailable: RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items.
I20240501 20:01:44.723626 29624 leader_election.cc:302] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:44.666484 31330 raft_consensus.cc:482] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:44.691640 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48630ae6eb394ac1b54955dd01cf0ed2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:44.689383 31153 raft_consensus.cc:482] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:44.734208 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcf30cf66714efdac997862dafbe4a2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
W20240501 20:01:44.739285 31341 log.cc:924] Time spent T d1d2cac25f354a97b1bfb53d57ae7c08 P ee0d97247d394a8facfa774f15f3a67d: Append to log took a long time: real 0.077s user 0.001s sys 0.000s
I20240501 20:01:44.669338 31201 consensus_queue.cc:1046] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.059s
I20240501 20:01:44.664935 31311 raft_consensus.cc:504] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:44.738262 29732 raft_consensus.cc:2454] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 4.
W20240501 20:01:44.744441 31334 log.cc:924] Time spent T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20240501 20:01:44.743497 29800 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6e38a2e91a3d48d09a22abce723af833"
dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
mode: GRACEFUL
new_leader_uuid: "fdfeac4190e045f290e003b2298d988d"
from {username='slave'} at 127.0.0.1:33582
I20240501 20:01:44.737434 31330 raft_consensus.cc:3037] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:44.748857 29800 raft_consensus.cc:595] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Received request to transfer leadership to TS fdfeac4190e045f290e003b2298d988d
W20240501 20:01:44.749675 29624 leader_election.cc:334] T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Remote error: Service unavailable: RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:36270 dropped due to backpressure. The service queue is full; it has 50 items.
I20240501 20:01:44.750241 29624 leader_election.cc:302] T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:44.751019 29800 raft_consensus.cc:603] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240501 20:01:44.640857 31245 consensus_queue.cc:260] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:44.730666 31333 raft_consensus.cc:2726] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:44.740552 31208 raft_consensus.cc:482] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:44.748442 31220 raft_consensus.cc:482] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:44.767328 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb27193abbf4644a48662909bc7b04a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
W20240501 20:01:44.771665 29625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 190975us
I20240501 20:01:44.777014 29625 leader_election.cc:302] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:44.771400 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0374bb790ed345ed90452595f71383a2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:44.755751 31201 consensus_queue.cc:1225] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer ee0d97247d394a8facfa774f15f3a67d log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20240501 20:01:44.756227 31170 raft_consensus.cc:2726] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240501 20:01:44.740917 31153 raft_consensus.cc:504] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:44.764389 31208 raft_consensus.cc:504] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:44.730113 31060 consensus_queue.cc:237] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:44.795382 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b337b8e3e38d4ad6b1076fbd0d0a1609" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:44.798802 31311 leader_election.cc:288] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:44.800285 31267 leader_election.cc:288] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:44.800860 29772 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.529s user 0.011s sys 0.021s
W20240501 20:01:44.801173 29772 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.529s user 0.011s sys 0.021s
W20240501 20:01:44.801425 29772 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 535569us
I20240501 20:01:44.789216 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcf30cf66714efdac997862dafbe4a2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:44.791198 29625 leader_election.cc:302] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:44.809686 31333 raft_consensus.cc:2726] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240501 20:01:44.810433 29625 leader_election.cc:302] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters: 888d99522e4844118f8f1cf26f268c8c
I20240501 20:01:44.811265 31311 raft_consensus.cc:2781] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240501 20:01:44.811619 31311 raft_consensus.cc:482] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Starting leader election (detected failure of leader 888d99522e4844118f8f1cf26f268c8c)
I20240501 20:01:44.811947 31311 raft_consensus.cc:3037] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:01:44.815950 29726 raft_consensus.cc:2454] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 5.
W20240501 20:01:44.817270 31282 consensus_meta.cc:287] T 1ca3948261374cef8b1b206a43c00173 P 4c8da1d580614449a636b855bf11b95f: Time spent flushing consensus metadata: real 0.899s user 0.010s sys 0.001s
I20240501 20:01:44.818755 29625 leader_election.cc:302] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:44.819429 31282 raft_consensus.cc:2781] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:44.819821 31282 raft_consensus.cc:482] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240501 20:01:44.820125 31043 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.927s user 0.013s sys 0.015s
W20240501 20:01:44.820798 31043 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.928s user 0.013s sys 0.015s
I20240501 20:01:44.820154 31282 raft_consensus.cc:3037] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:44.820943 29625 leader_election.cc:302] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:44.827092 29772 leader_election.cc:302] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
W20240501 20:01:44.780133 30383 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 125733us
W20240501 20:01:44.830027 29735 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 815) took 4024 ms (client timeout 2129 ms). Trace:
W20240501 20:01:44.837623 29735 rpcz_store.cc:269] 0501 20:01:40.806037 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:44.829805 (+4023768us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:44.829947 (+ 142us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240501 20:01:44.830528 29653 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22"
dest_uuid: "4c8da1d580614449a636b855bf11b95f"
mode: GRACEFUL
new_leader_uuid: "fdfeac4190e045f290e003b2298d988d"
from {username='slave'} at 127.0.0.1:60736
I20240501 20:01:44.779381 31220 raft_consensus.cc:504] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:44.779054 31344 raft_consensus.cc:2726] T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:44.838481 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38cbeb0dc0c14e3d9182101eea885625" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:44.779042 31181 consensus_queue.cc:1046] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.046s
I20240501 20:01:44.793422 31336 raft_consensus.cc:2726] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:44.842818 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:44.838809 29653 raft_consensus.cc:595] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Received request to transfer leadership to TS fdfeac4190e045f290e003b2298d988d
I20240501 20:01:44.845530 29653 raft_consensus.cc:603] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240501 20:01:44.839278 29728 raft_consensus.cc:3037] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Advancing to term 5
W20240501 20:01:44.843720 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 85745.5us
I20240501 20:01:44.809213 31245 raft_consensus.cc:3037] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:44.847340 29735 raft_consensus.cc:3037] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:44.786355 31140 consensus_queue.cc:1046] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.054s
I20240501 20:01:44.829492 31267 raft_consensus.cc:2781] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election won for term 5
W20240501 20:01:44.837961 31258 consensus_meta.cc:287] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f: Time spent flushing consensus metadata: real 0.524s user 0.008s sys 0.000s
I20240501 20:01:44.857597 29772 leader_election.cc:302] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:44.845206 31212 leader_election.cc:288] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:44.861214 31208 leader_election.cc:288] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:44.838104 31170 consensus_queue.cc:1035] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20240501 20:01:44.858769 29728 raft_consensus.cc:2454] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 5.
I20240501 20:01:44.863596 29732 raft_consensus.cc:2454] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 4.
I20240501 20:01:44.852757 31267 raft_consensus.cc:482] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:44.864614 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5525f546426c47a38edf7f56ad04b7ab" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
W20240501 20:01:44.865322 31270 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20240501 20:01:44.862057 31132 tablet_copy_client.cc:523] T be383be783234855878cb8fd442c7b05 P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:01:44.867619 31267 raft_consensus.cc:3037] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:44.858211 29802 raft_consensus.cc:1261] T 0fb27193abbf4644a48662909bc7b04a P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240501 20:01:44.885691 31153 leader_election.cc:288] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:44.886870 31343 log.cc:924] Time spent T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.110s user 0.000s sys 0.001s
I20240501 20:01:44.853109 31360 raft_consensus.cc:2781] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240501 20:01:44.888824 31340 raft_consensus.cc:2781] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:44.977263 31340 raft_consensus.cc:482] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:44.977653 31340 raft_consensus.cc:3037] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Advancing to term 3
W20240501 20:01:44.978672 31358 log.cc:924] Time spent T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.159s user 0.000s sys 0.001s
I20240501 20:01:44.979372 31336 raft_consensus.cc:3032] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:01:44.979732 31336 raft_consensus.cc:729] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:44.980475 31336 consensus_queue.cc:260] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:44.981511 31336 raft_consensus.cc:3037] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:44.982755 31330 raft_consensus.cc:504] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:44.985109 31311 raft_consensus.cc:504] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:44.985416 31282 raft_consensus.cc:504] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:44.986960 31311 leader_election.cc:288] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 7 election: Requested vote from peers ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:44.988565 31246 raft_consensus.cc:3032] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:01:44.988946 31246 raft_consensus.cc:729] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:44.989679 31246 consensus_queue.cc:260] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:44.988397 31359 log.cc:924] Time spent T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 0.107s user 0.003s sys 0.000s
I20240501 20:01:44.990396 31282 leader_election.cc:288] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:44.990733 31246 raft_consensus.cc:3037] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:44.974403 31360 raft_consensus.cc:482] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:45.004688 31042 raft_consensus.cc:3037] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:45.006513 31330 leader_election.cc:288] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:45.007822 30906 raft_consensus.cc:3032] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:01:45.008193 30906 raft_consensus.cc:729] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:45.014281 31340 raft_consensus.cc:504] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:45.015653 31340 leader_election.cc:288] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:45.002293 31267 raft_consensus.cc:504] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:45.021785 31360 raft_consensus.cc:3037] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Advancing to term 5
W20240501 20:01:45.052788 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 198004us
I20240501 20:01:45.065368 29727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 965) took 1130 ms. Trace:
I20240501 20:01:45.065234 29657 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9f261d2a05bf4c838532049601694b4a"
dest_uuid: "4c8da1d580614449a636b855bf11b95f"
mode: GRACEFUL
new_leader_uuid: "fdfeac4190e045f290e003b2298d988d"
from {username='slave'} at 127.0.0.1:60736
I20240501 20:01:45.066197 29732 raft_consensus.cc:2396] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20240501 20:01:45.066193 29727 rpcz_store.cc:276] 0501 20:01:43.934964 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:44.337844 (+402880us) service_pool.cc:225] Handling call
0501 20:01:44.471561 (+133717us) raft_consensus.cc:1457] Updating replica for 1 ops
0501 20:01:44.591913 (+120352us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:44.593663 (+ 1750us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0501 20:01:44.718426 (+124763us) log.cc:842] Serialized 26 byte log entry
0501 20:01:44.849653 (+131227us) raft_consensus.cc:1631] Marking committed up to 1
0501 20:01:44.853520 (+ 3867us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:44.860165 (+ 6645us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0501 20:01:45.064603 (+204438us) raft_consensus.cc:1671] finished
0501 20:01:45.064656 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:45.065078 (+ 422us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":79365,"raft.queue_time_us":98143,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":5579,"thread_start_us":100495,"threads_started":1,"wal-append.queue_time_us":206866}
I20240501 20:01:45.070773 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5345f6eb62449ba69b4a9c2e349b31" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:45.072909 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:45.074819 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfac893b4b4f457694a4eb184b6abb53" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:45.076028 29727 raft_consensus.cc:2396] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20240501 20:01:45.082022 29735 raft_consensus.cc:2454] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 5.
I20240501 20:01:45.086540 30414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e89c0572e4ee7be8e5315b459bdb3" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c"
I20240501 20:01:45.077195 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c597f15e844f30837012baed4b0501" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:45.083787 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5345f6eb62449ba69b4a9c2e349b31" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:45.022343 31267 leader_election.cc:288] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:45.100742 29660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59136 (request call id 824) took 2689 ms. Trace:
I20240501 20:01:45.105036 29624 leader_election.cc:302] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: 888d99522e4844118f8f1cf26f268c8c
W20240501 20:01:45.105932 29726 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 842) took 1772 ms (client timeout 1727 ms). Trace:
I20240501 20:01:45.106631 29653 raft_consensus.cc:1226] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 7 FOLLOWER]: Rejecting Update request from peer 888d99522e4844118f8f1cf26f268c8c for earlier term 6. Current term is 7. Ops: []
I20240501 20:01:45.108538 29807 raft_consensus.cc:1226] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [term 5 LEADER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 4. Current term is 5. Ops: []
W20240501 20:01:45.109103 29729 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 902) took 4223 ms (client timeout 2035 ms). Trace:
I20240501 20:01:45.095039 29657 raft_consensus.cc:595] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Received request to transfer leadership to TS fdfeac4190e045f290e003b2298d988d
W20240501 20:01:45.117723 29729 rpcz_store.cc:269] 0501 20:01:40.899904 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:45.108584 (+4208680us) service_pool.cc:212] Skipping call since client already timed out
0501 20:01:45.108804 (+ 220us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240501 20:01:45.121243 31205 raft_consensus.cc:3032] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Stepping down as leader of term 4
I20240501 20:01:45.101408 29660 rpcz_store.cc:276] 0501 20:01:42.410767 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:42.477991 (+ 67224us) service_pool.cc:225] Handling call
0501 20:01:42.560991 (+ 83000us) raft_consensus.cc:1457] Updating replica for 1 ops
0501 20:01:42.571353 (+ 10362us) raft_consensus.cc:1520] Early marking committed up to index 2
0501 20:01:42.586550 (+ 15197us) log.cc:842] Serialized 14 byte log entry
0501 20:01:43.590316 (+1003766us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0501 20:01:43.590812 (+ 496us) log.cc:842] Serialized 26 byte log entry
0501 20:01:43.590943 (+ 131us) raft_consensus.cc:1631] Marking committed up to 2
0501 20:01:43.590979 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:43.591088 (+ 109us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0501 20:01:43.864719 (+273631us) spinlock_profiling.cc:243] Waited 23.6 ms on lock 0x7b44000232cc. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe454797b51 00007fe454865685 00007fe4548660c2 00007fe45750e441 00007fe457509d4a 00007fe4574fdb1e 00007fe4574fce66 00007fe4574fca88 00007fe457506481 00007fe457506439 00007fe4575063c9
0501 20:01:43.866125 (+ 1406us) raft_consensus.cc:1671] finished
0501 20:01:43.866240 (+ 115us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:45.100195 (+1233955us) spinlock_profiling.cc:243] Waited 76.7 ms on lock 0x7b54001a27d0. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787ac59 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc 00007fe455e9e782 00007fe455ca2bcc 00007fe455ca2006 00007fe455ca19af
0501 20:01:45.100489 (+ 294us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":88073,"prepare.queue_time_us":15440,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":204,"raft.run_cpu_time_us":286,"raft.run_wall_time_us":16027,"spinlock_wait_cycles":230830720,"thread_start_us":1003305,"threads_started":1,"wal-append.queue_time_us":1251471,"wal-append.run_cpu_time_us":4584,"wal-append.run_wall_time_us":1028688}
I20240501 20:01:45.121703 31205 raft_consensus.cc:729] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:45.125077 29657 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a8e9f75387564275a60d38e9a910319c"
dest_uuid: "4c8da1d580614449a636b855bf11b95f"
mode: GRACEFUL
new_leader_uuid: "fdfeac4190e045f290e003b2298d988d"
from {username='slave'} at 127.0.0.1:60736
I20240501 20:01:45.125977 29657 raft_consensus.cc:595] T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f [term 2 LEADER]: Received request to transfer leadership to TS fdfeac4190e045f290e003b2298d988d
I20240501 20:01:45.127815 29651 raft_consensus.cc:1226] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 2. Current term is 3. Ops: []
I20240501 20:01:45.128736 31220 leader_election.cc:288] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:45.124441 31205 consensus_queue.cc:260] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:45.131314 31205 raft_consensus.cc:3037] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:45.136482 31279 raft_consensus.cc:2781] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election won for term 4
W20240501 20:01:45.106647 29772 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 249002us
W20240501 20:01:45.110116 29726 rpcz_store.cc:269] 0501 20:01:43.333829 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:44.271390 (+937561us) service_pool.cc:225] Handling call
0501 20:01:45.105489 (+834099us) spinlock_profiling.cc:243] Waited 525 ms on lock 0x7b54001ef5d4. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787f4fe 00007fe45c7bd9d7 00007fe455ea1c14 00007fe455ea1ba2 00007fe455ea1ad1 00007fe455ea1a4c 00007fe455ea0d02 00007fe455ca2bcc 00007fe455ca2006 00007fe455ca19af
0501 20:01:45.105697 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1207196416}
I20240501 20:01:45.136977 31279 raft_consensus.cc:482] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 888d99522e4844118f8f1cf26f268c8c)
I20240501 20:01:45.146337 29731 raft_consensus.cc:2396] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20240501 20:01:45.146101 31311 raft_consensus.cc:482] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:45.153934 29651 raft_consensus.cc:1226] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 3. Current term is 4. Ops: []
I20240501 20:01:45.148269 30944 consensus_queue.cc:1046] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.010s
I20240501 20:01:45.158138 30944 raft_consensus.cc:3032] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 6 LEADER]: Stepping down as leader of term 6
I20240501 20:01:45.158725 30944 raft_consensus.cc:729] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 888d99522e4844118f8f1cf26f268c8c, State: Running, Role: LEADER
I20240501 20:01:45.151989 29653 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aa8cd40cb4044d18a86001ce6ddf97ab"
dest_uuid: "4c8da1d580614449a636b855bf11b95f"
mode: GRACEFUL
new_leader_uuid: "fdfeac4190e045f290e003b2298d988d"
from {username='slave'} at 127.0.0.1:60736
I20240501 20:01:45.163650 29734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 970) took 1225 ms. Trace:
I20240501 20:01:45.163478 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c597f15e844f30837012baed4b0501" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:45.164317 29734 rpcz_store.cc:276] 0501 20:01:43.938425 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:44.335992 (+397567us) service_pool.cc:225] Handling call
0501 20:01:44.336306 (+ 314us) raft_consensus.cc:1457] Updating replica for 1 ops
0501 20:01:44.357760 (+ 21454us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:44.357814 (+ 54us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0501 20:01:44.379389 (+ 21575us) log.cc:842] Serialized 26 byte log entry
0501 20:01:44.690517 (+311128us) raft_consensus.cc:1631] Marking committed up to 2
0501 20:01:44.785789 (+ 95272us) log.cc:842] Serialized 14 byte log entry
0501 20:01:45.159110 (+373321us) spinlock_profiling.cc:243] Waited 30.2 ms on lock 0x7b18001b1de0. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe45d7f2661 00007fe45d7ff3e1 00007fe4548ca2ea 00007fe45dfb9fe5 00007fe45494e3d4 00007fe45494dcc5 00007fe457873620 00007fe4578857c6 00007fe457896327 00007fe4578962a8 00007fe457896228
0501 20:01:45.159627 (+ 517us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:45.159802 (+ 175us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0501 20:01:45.159892 (+ 90us) raft_consensus.cc:1671] finished
0501 20:01:45.160019 (+ 127us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:45.163449 (+ 3430us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":47777,"prepare.queue_time_us":94723,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":9574,"raft.queue_time_us":414602,"raft.run_cpu_time_us":390,"raft.run_wall_time_us":91548,"spinlock_wait_cycles":69461760,"thread_start_us":260105,"threads_started":2,"wal-append.queue_time_us":272220}
I20240501 20:01:45.166891 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a35893b91247a99912eba787d787cf" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:45.160570 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668b4a4d782c4278932dd8413dc3ab3e" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
W20240501 20:01:45.168377 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 365657us
I20240501 20:01:45.174613 31043 raft_consensus.cc:3032] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:01:45.174727 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "456035a1cb3f4f7da52d0fc1dd1c656c" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:45.175143 31043 raft_consensus.cc:729] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:45.175069 31143 consensus_queue.cc:1035] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 20:01:45.177209 29804 raft_consensus.cc:1226] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 4 LEADER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 3. Current term is 4. Ops: []
I20240501 20:01:45.178668 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8edcd903224d48f889b46e1b5318e0ec" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:45.180311 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49da4246f5f6438abe0e7e8c9a523d80" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:45.181102 29800 raft_consensus.cc:3037] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:45.182478 31181 consensus_queue.cc:1225] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer ee0d97247d394a8facfa774f15f3a67d log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20240501 20:01:45.182828 29558 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33356 (request call id 51) took 7714 ms. Trace:
I20240501 20:01:45.184223 29653 raft_consensus.cc:595] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Received request to transfer leadership to TS fdfeac4190e045f290e003b2298d988d
I20240501 20:01:45.185845 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcf30cf66714efdac997862dafbe4a2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:45.147346 31311 raft_consensus.cc:504] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:45.186758 29807 raft_consensus.cc:2379] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d in current term 5: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:45.188107 30944 consensus_queue.cc:260] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
W20240501 20:01:45.190064 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 46466us
I20240501 20:01:45.187705 30906 consensus_queue.cc:260] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:45.190659 28519 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 6
I20240501 20:01:45.191560 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac2171683884f6f97cdfeb0a656cb49" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:45.192497 29731 raft_consensus.cc:2362] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 4. Re-sending same reply.
I20240501 20:01:45.193578 30944 raft_consensus.cc:3037] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:01:45.193847 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca3948261374cef8b1b206a43c00173" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:45.195595 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcf30cf66714efdac997862dafbe4a2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:45.197321 29731 raft_consensus.cc:2354] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d for earlier term 4. Current term is 5.
I20240501 20:01:45.198349 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3f29f0eeea4b119984634202e5d9cd" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:45.198604 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece8d4ed84c34b748f7b6272b31c90ac" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:45.199294 29731 raft_consensus.cc:2379] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 4: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:45.191589 30906 raft_consensus.cc:3037] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:45.200240 29657 raft_consensus.cc:1226] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 4. Current term is 5. Ops: []
I20240501 20:01:45.202008 29651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211e27a3b0084b89a6dd7550849b5a65" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:45.148563 31279 raft_consensus.cc:3037] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Advancing to term 4
W20240501 20:01:45.197338 29733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36260 (request call id 850) took 1580 ms (client timeout 1640 ms). Trace:
W20240501 20:01:45.208516 29733 rpcz_store.cc:269] 0501 20:01:43.617138 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:44.272895 (+655757us) service_pool.cc:225] Handling call
0501 20:01:45.196982 (+924087us) spinlock_profiling.cc:243] Waited 12.7 ms on lock 0x7b54001013d4. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787f4fe 00007fe45c7bd9d7 00007fe455ea1c14 00007fe455ea1ba2 00007fe455ea1ad1 00007fe455ea1a4c 00007fe455ea0d02 00007fe455ca2bcc 00007fe455ca2006 00007fe455ca19af
0501 20:01:45.197162 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":29182720}
I20240501 20:01:45.208940 31311 leader_election.cc:288] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:45.209160 29726 raft_consensus.cc:2454] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 5.
I20240501 20:01:45.209755 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d86f8848a184a87947fd927c44e5a35" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:45.209910 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfac893b4b4f457694a4eb184b6abb53" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:45.210822 29728 raft_consensus.cc:2396] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20240501 20:01:45.211052 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668b4a4d782c4278932dd8413dc3ab3e" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:45.212085 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36057b3f24d44b9aac3a1378053a395f" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:45.212931 29732 raft_consensus.cc:2454] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 3.
I20240501 20:01:45.213302 29624 leader_election.cc:302] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: 888d99522e4844118f8f1cf26f268c8c
I20240501 20:01:45.212942 31135 consensus_queue.cc:1035] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.002s
I20240501 20:01:45.215020 31233 raft_consensus.cc:3032] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Stepping down as leader of term 4
I20240501 20:01:45.215556 31233 raft_consensus.cc:729] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:45.216787 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69915dd5992641e3a24f7f45f1cd5c0c" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:45.217057 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e9f75387564275a60d38e9a910319c" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:45.217607 29730 raft_consensus.cc:2454] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 4.
I20240501 20:01:45.218757 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "696121417ea9403990456cfc6fc4f416" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:45.219489 31043 consensus_queue.cc:260] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:45.221246 29772 leader_election.cc:302] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:45.221958 31043 raft_consensus.cc:3037] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:45.222359 31118 raft_consensus.cc:2781] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:01:45.223058 31118 raft_consensus.cc:686] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 5 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:45.224400 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38cbeb0dc0c14e3d9182101eea885625" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:45.225210 29732 raft_consensus.cc:2362] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 4. Re-sending same reply.
I20240501 20:01:45.227571 31063 raft_consensus.cc:2781] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240501 20:01:45.228062 31063 raft_consensus.cc:482] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting leader election (detected failure of leader 888d99522e4844118f8f1cf26f268c8c)
I20240501 20:01:45.228557 31063 raft_consensus.cc:3037] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:45.227733 31118 consensus_queue.cc:237] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:45.220301 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010ceabf3e5a4b1c9330b80dade765bb" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:45.219964 31233 consensus_queue.cc:260] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:45.211661 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07bdb8b09b8c4ef4b8e49daa9b8c01d6" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:45.218725 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5525f546426c47a38edf7f56ad04b7ab" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:45.200242 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a35893b91247a99912eba787d787cf" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:45.233330 29734 raft_consensus.cc:2454] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 4.
I20240501 20:01:45.234700 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9472c53f79ec423f82c396bd980674e2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:45.169019 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1a08b455b94ae089396a15c837ddc2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:45.238365 28519 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:01:45.240662 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "696121417ea9403990456cfc6fc4f416" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:45.242195 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649b71f9f267437ca51c758dd9ca6987" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:45.242136 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48fd0a8e289401aa7417c42ff2c81b2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:45.250654 29732 raft_consensus.cc:2396] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240501 20:01:45.251494 29651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3f29f0eeea4b119984634202e5d9cd" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:45.252430 29651 raft_consensus.cc:2379] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d in current term 4: Already voted for candidate ee0d97247d394a8facfa774f15f3a67d in this term.
I20240501 20:01:45.250084 29729 raft_consensus.cc:2396] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20240501 20:01:45.253679 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d86f8848a184a87947fd927c44e5a35" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:45.254633 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f5d1686214485d91a0baf4b870740e" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:45.255034 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e49dd5180f04313ab33942fb3ddc147" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:45.255494 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e49dd5180f04313ab33942fb3ddc147" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:45.263873 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458740ab5c444b6ab3d1991c69db5084" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:45.264920 29729 raft_consensus.cc:2379] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 4: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:45.258963 29800 raft_consensus.cc:2454] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 6.
I20240501 20:01:45.266710 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77be7101c0fe4085b6706296f609870a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:45.265944 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af08384057a4d85977481472f460168" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:45.259543 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e38a2e91a3d48d09a22abce723af833" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:45.268919 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcf30cf66714efdac997862dafbe4a2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:45.270736 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca3948261374cef8b1b206a43c00173" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:45.270855 29651 raft_consensus.cc:1226] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 4. Current term is 5. Ops: []
I20240501 20:01:45.267927 31212 consensus_queue.cc:1035] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20240501 20:01:45.275079 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:01:45.276553 31233 raft_consensus.cc:3037] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:45.183310 29558 rpcz_store.cc:276] 0501 20:01:37.468669 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:37.473053 (+ 4384us) service_pool.cc:225] Handling call
0501 20:01:45.182563 (+7709510us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0501 20:01:39.348609 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 20:01:39.464684 (+116075us) write_op.cc:424] Acquiring schema lock in shared mode
0501 20:01:39.464721 (+ 37us) write_op.cc:427] Acquired schema lock
0501 20:01:39.464738 (+ 17us) tablet.cc:660] Decoding operations
0501 20:01:39.465173 (+ 435us) write_op.cc:612] Acquiring the partition lock for write op
0501 20:01:39.465226 (+ 53us) write_op.cc:633] Partition lock acquired for write op
0501 20:01:39.465254 (+ 28us) tablet.cc:683] Acquiring locks for 1 operations
0501 20:01:39.465436 (+ 182us) tablet.cc:699] Row locks acquired
0501 20:01:39.465454 (+ 18us) write_op.cc:260] PREPARE: finished
0501 20:01:39.465576 (+ 122us) write_op.cc:270] Start()
0501 20:01:39.465677 (+ 101us) write_op.cc:276] Timestamp: P: 1714593699465534 usec, L: 0
0501 20:01:39.465696 (+ 19us) op_driver.cc:331] REPLICATION: starting
0501 20:01:39.466306 (+ 610us) log.cc:842] Serialized 538 byte log entry
0501 20:01:42.354033 (+2887727us) spinlock_profiling.cc:243] Waited 4.14 ms on lock 0x7b44000232cc. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe454797b51 00007fe454865685 00007fe4548660c2 00007fe45750e441 00007fe457509d4a 00007fe4574fdb1e 00007fe4574fce66 00007fe4574fca88 00007fe457506481 00007fe457506439 00007fe4575063c9
0501 20:01:43.766253 (+1412220us) op_driver.cc:444] REPLICATION: finished
0501 20:01:43.885005 (+118752us) spinlock_profiling.cc:243] Waited 1.54 ms on lock 0x7b44000232cc. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe454797b51 00007fe454865685 00007fe4548660c2 00007fe45750e441 00007fe457509d4a 00007fe45493316e 00007fe45d76c939 00007fe45495042b 00007fe45494e82f 00007fe454950992 00007fe45b28cda0
0501 20:01:44.318209 (+433204us) write_op.cc:301] APPLY: starting
0501 20:01:44.327140 (+ 8931us) tablet.cc:1365] starting BulkCheckPresence
0501 20:01:44.346242 (+ 19102us) tablet.cc:1368] finished BulkCheckPresence
0501 20:01:44.352711 (+ 6469us) tablet.cc:1370] starting ApplyRowOperation cycle
0501 20:01:44.418149 (+ 65438us) tablet.cc:1381] finished ApplyRowOperation cycle
0501 20:01:44.485874 (+ 67725us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0501 20:01:44.491368 (+ 5494us) write_op.cc:312] APPLY: finished
0501 20:01:44.511300 (+ 19932us) log.cc:842] Serialized 22 byte log entry
0501 20:01:44.581223 (+ 69923us) write_op.cc:481] Releasing partition, row and schema locks
0501 20:01:44.590734 (+ 9511us) write_op.cc:446] Released schema lock
0501 20:01:44.592870 (+ 2136us) write_op.cc:341] FINISH: Updating metrics
0501 20:01:44.662325 (+ 69455us) write_op.cc:481] Releasing partition, row and schema locks
0501 20:01:44.666127 (+ 3802us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":15175,"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":514985,"apply.run_cpu_time_us":7330,"apply.run_wall_time_us":379092,"num_ops":1,"prepare.queue_time_us":840422,"prepare.run_cpu_time_us":36640,"prepare.run_wall_time_us":758122,"raft.queue_time_us":1328258,"raft.run_cpu_time_us":91142,"raft.run_wall_time_us":463407,"replication_time_us":4292413,"spinlock_wait_cycles":13048960,"thread_start_us":2737697,"threads_started":4,"wal-append.queue_time_us":787182}]]}
I20240501 20:01:45.283285 30384 leader_election.cc:302] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 888d99522e4844118f8f1cf26f268c8c; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:45.284492 31292 raft_consensus.cc:2726] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 20:01:45.266872 29624 leader_election.cc:302] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:45.288954 29732 raft_consensus.cc:2354] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d for earlier term 4. Current term is 5.
I20240501 20:01:45.297384 31132 tablet_bootstrap.cc:492] T be383be783234855878cb8fd442c7b05 P 888d99522e4844118f8f1cf26f268c8c: Bootstrap starting.
I20240501 20:01:45.301611 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb27193abbf4644a48662909bc7b04a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:45.304010 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a35893b91247a99912eba787d787cf" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:45.302242 29657 raft_consensus.cc:2454] T 1ca3948261374cef8b1b206a43c00173 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 2.
I20240501 20:01:45.300384 29731 raft_consensus.cc:2454] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 5.
I20240501 20:01:45.306056 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05df43bdfa479b902b1da297fb5974" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:45.307114 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d86f8848a184a87947fd927c44e5a35" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:45.307803 29700 leader_election.cc:302] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:45.308422 31367 consensus_queue.cc:1046] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 20:01:45.309870 31367 consensus_queue.cc:1225] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer ee0d97247d394a8facfa774f15f3a67d log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20240501 20:01:45.312348 31087 raft_consensus.cc:2781] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Leader election won for term 6
I20240501 20:01:45.312861 31087 raft_consensus.cc:686] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [term 6 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:45.312896 29806 raft_consensus.cc:1203] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [term 4 LEADER]: Deduplicated request from leader. Original: 2.2->[3.3-3.3] Dedup: 3.3->[]
I20240501 20:01:45.313602 29806 raft_consensus.cc:1226] T 2d86f8848a184a87947fd927c44e5a35 P ee0d97247d394a8facfa774f15f3a67d [term 4 LEADER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20240501 20:01:45.313817 31063 raft_consensus.cc:504] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:45.314653 31175 raft_consensus.cc:3032] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Stepping down as leader of term 4
I20240501 20:01:45.315062 31175 raft_consensus.cc:729] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:45.316021 31175 consensus_queue.cc:260] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:45.317399 31175 raft_consensus.cc:3037] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:45.318272 29729 raft_consensus.cc:1261] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:01:45.318905 29804 raft_consensus.cc:1261] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240501 20:01:45.321059 30414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "696121417ea9403990456cfc6fc4f416" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c" is_pre_election: true
I20240501 20:01:45.322032 30414 raft_consensus.cc:2379] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 4: Already voted for candidate 888d99522e4844118f8f1cf26f268c8c in this term.
I20240501 20:01:45.314675 29808 raft_consensus.cc:1261] T 1ca3948261374cef8b1b206a43c00173 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:01:45.326270 29651 raft_consensus.cc:1226] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20240501 20:01:45.314155 31087 consensus_queue.cc:237] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:45.327189 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 87501.1us
I20240501 20:01:45.319907 29729 raft_consensus.cc:1261] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240501 20:01:45.309118 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010ceabf3e5a4b1c9330b80dade765bb" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:45.327977 31205 consensus_queue.cc:1035] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:45.332167 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:45.299355 31360 raft_consensus.cc:504] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:45.329015 31151 consensus_queue.cc:1035] T 1ca3948261374cef8b1b206a43c00173 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20240501 20:01:45.337380 29730 raft_consensus.cc:2379] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 3: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:45.342629 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458740ab5c444b6ab3d1991c69db5084" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:45.303841 31135 raft_consensus.cc:482] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:45.341876 29726 raft_consensus.cc:2362] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 4. Re-sending same reply.
I20240501 20:01:45.333560 31063 leader_election.cc:288] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:45.349931 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6797d82dda2948d8a7445fcd33e44752" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:45.308799 31212 consensus_queue.cc:1046] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20240501 20:01:45.354223 29802 raft_consensus.cc:1226] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 4. Current term is 5. Ops: []
I20240501 20:01:45.354630 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be383be783234855878cb8fd442c7b05" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:45.354691 30414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd5e83faaff4e2da4e4cc22ba410d33" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c"
I20240501 20:01:45.363509 31212 consensus_queue.cc:1225] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer ee0d97247d394a8facfa774f15f3a67d log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20240501 20:01:45.364194 31212 raft_consensus.cc:3032] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:01:45.364359 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77be7101c0fe4085b6706296f609870a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:45.365294 29735 raft_consensus.cc:2379] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 3: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:45.364604 31212 raft_consensus.cc:729] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:45.369926 29732 raft_consensus.cc:2379] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 4: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:45.374492 31279 raft_consensus.cc:504] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:45.389864 29726 raft_consensus.cc:1261] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20240501 20:01:45.398128 31197 raft_consensus.cc:482] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:45.407428 31246 consensus_queue.cc:1046] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 20:01:45.360878 31135 raft_consensus.cc:504] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:45.408536 31279 leader_election.cc:288] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:45.408703 31246 raft_consensus.cc:3032] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Stepping down as leader of term 4
I20240501 20:01:45.409142 31246 raft_consensus.cc:729] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:45.410264 31135 leader_election.cc:288] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:45.410159 31246 consensus_queue.cc:260] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:45.411403 31246 raft_consensus.cc:3037] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:45.412107 29653 raft_consensus.cc:1226] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Rejecting Update request from peer 888d99522e4844118f8f1cf26f268c8c for earlier term 3. Current term is 4. Ops: []
I20240501 20:01:45.413412 30414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e07220db7444ae97f294d70eda6a93" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c"
I20240501 20:01:45.400947 29726 pending_rounds.cc:76] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d: Aborting all ops after (but not including) 2
I20240501 20:01:45.419348 29731 raft_consensus.cc:2454] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 5.
I20240501 20:01:45.423625 31212 consensus_queue.cc:260] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:45.424938 31360 leader_election.cc:288] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:45.425508 31212 raft_consensus.cc:3037] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:45.426707 29728 raft_consensus.cc:2454] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 5.
I20240501 20:01:45.428431 29729 raft_consensus.cc:1261] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240501 20:01:45.428845 31292 consensus_queue.cc:1046] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:01:45.429519 31221 raft_consensus.cc:482] T 84fa28fb55f44b61ba1a88ed775a78ba P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:45.429905 31292 raft_consensus.cc:3032] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:01:45.441272 29624 leader_election.cc:302] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:45.440963 29556 catalog_manager.cc:5484] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 2 to 3, leader changed from ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 3 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:45.442764 29624 leader_election.cc:302] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:45.439860 31292 raft_consensus.cc:729] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 888d99522e4844118f8f1cf26f268c8c, State: Running, Role: LEADER
I20240501 20:01:45.444025 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac2171683884f6f97cdfeb0a656cb49" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:45.444224 31292 consensus_queue.cc:260] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:45.439864 29652 raft_consensus.cc:1226] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 3. Current term is 4. Ops: []
I20240501 20:01:45.409161 31197 raft_consensus.cc:504] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:45.445858 31333 raft_consensus.cc:2726] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 20:01:45.444903 29653 raft_consensus.cc:2379] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d in current term 4: Already voted for candidate ee0d97247d394a8facfa774f15f3a67d in this term.
I20240501 20:01:45.417196 29726 pending_rounds.cc:97] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20240501 20:01:45.448822 29726 raft_consensus.cc:2866] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 20:01:45.443189 29556 catalog_manager.cc:5484] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 1 to 3, leader changed from ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 3 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:45.455003 31118 raft_consensus.cc:482] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:45.456854 31153 raft_consensus.cc:2726] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240501 20:01:45.457319 31267 raft_consensus.cc:482] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:45.457393 31118 raft_consensus.cc:504] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:45.458038 31267 raft_consensus.cc:504] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:45.459753 29773 leader_election.cc:334] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 1.523s (SENT)
I20240501 20:01:45.459779 31118 leader_election.cc:288] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:45.464956 29660 raft_consensus.cc:2379] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d in current term 4: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:45.464720 29556 catalog_manager.cc:5484] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:45.469574 29556 catalog_manager.cc:5484] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:45.447888 31063 raft_consensus.cc:2726] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 20:01:45.476740 29652 raft_consensus.cc:1226] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Rejecting Update request from peer 888d99522e4844118f8f1cf26f268c8c for earlier term 5. Current term is 6. Ops: []
I20240501 20:01:45.451542 29657 raft_consensus.cc:2454] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 4.
I20240501 20:01:45.488608 29734 raft_consensus.cc:2454] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 4.
I20240501 20:01:45.494220 29772 leader_election.cc:302] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, 888d99522e4844118f8f1cf26f268c8c
W20240501 20:01:45.498855 31366 log.cc:924] Time spent T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d: Append to log took a long time: real 0.121s user 0.001s sys 0.000s
I20240501 20:01:45.499686 29653 raft_consensus.cc:1226] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 4. Current term is 5. Ops: []
I20240501 20:01:45.430311 31221 raft_consensus.cc:504] T 84fa28fb55f44b61ba1a88ed775a78ba P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:45.502192 31221 leader_election.cc:288] T 84fa28fb55f44b61ba1a88ed775a78ba P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:45.499603 31307 raft_consensus.cc:2726] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 20:01:45.504683 29732 raft_consensus.cc:1261] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240501 20:01:45.505395 31312 raft_consensus.cc:482] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:45.506192 31312 raft_consensus.cc:504] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:45.508337 31369 log.cc:924] Time spent T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d: Append to log took a long time: real 0.086s user 0.001s sys 0.000s
I20240501 20:01:45.508903 31292 raft_consensus.cc:3037] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Advancing to term 4
W20240501 20:01:45.509871 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 67186.8us
I20240501 20:01:45.476907 31267 leader_election.cc:288] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:45.517694 31312 leader_election.cc:288] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:45.518417 31223 consensus_queue.cc:1046] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240501 20:01:45.519657 31353 raft_consensus.cc:3032] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [term 5 LEADER]: Stepping down as leader of term 5
I20240501 20:01:45.519995 31353 raft_consensus.cc:729] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 888d99522e4844118f8f1cf26f268c8c, State: Running, Role: LEADER
I20240501 20:01:45.516736 29556 catalog_manager.cc:5484] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:45.520845 31353 consensus_queue.cc:260] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:45.497673 31197 leader_election.cc:288] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:45.524467 29731 raft_consensus.cc:1261] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240501 20:01:45.516149 29727 raft_consensus.cc:2454] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 4.
W20240501 20:01:45.530002 29773 leader_election.cc:334] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 1.564s (SENT)
I20240501 20:01:45.533735 29624 leader_election.cc:302] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:45.534046 29772 leader_election.cc:302] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, 888d99522e4844118f8f1cf26f268c8c
I20240501 20:01:45.536242 29733 raft_consensus.cc:2454] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 4.
I20240501 20:01:45.539059 31353 raft_consensus.cc:3037] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [term 5 FOLLOWER]: Advancing to term 6
W20240501 20:01:45.539237 29606 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 0374bb790ed345ed90452595f71383a2, TS ee0d97247d394a8facfa774f15f3a67d -> TS 888d99522e4844118f8f1cf26f268c8c move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:45.539585 29606 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 0374bb790ed345ed90452595f71383a2, TS ee0d97247d394a8facfa774f15f3a67d -> TS 888d99522e4844118f8f1cf26f268c8c move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:45.532996 29556 catalog_manager.cc:5484] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 1 to 4. New cstate: current_term: 4 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:45.533618 31212 raft_consensus.cc:482] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:45.539158 31223 raft_consensus.cc:482] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:45.567660 29624 leader_election.cc:302] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:45.569554 29624 leader_election.cc:302] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:45.571041 31333 raft_consensus.cc:2726] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:45.571822 31333 raft_consensus.cc:2726] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240501 20:01:45.573288 29624 leader_election.cc:302] T 2d86f8848a184a87947fd927c44e5a35 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:45.574090 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5345f6eb62449ba69b4a9c2e349b31" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:45.574872 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac2171683884f6f97cdfeb0a656cb49" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:45.580418 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 52351.7us
I20240501 20:01:45.561476 31212 raft_consensus.cc:504] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:45.558005 31267 raft_consensus.cc:2781] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240501 20:01:45.561241 31197 raft_consensus.cc:2726] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:01:45.583319 31223 raft_consensus.cc:504] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:45.606024 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84fa28fb55f44b61ba1a88ed775a78ba" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:45.566339 29556 catalog_manager.cc:5484] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 3 to 4, leader changed from 4c8da1d580614449a636b855bf11b95f (127.27.217.193) to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 4 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:45.607916 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77be7101c0fe4085b6706296f609870a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:45.608824 29660 raft_consensus.cc:2379] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d in current term 4: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:45.609323 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9472c53f79ec423f82c396bd980674e2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:45.609581 31063 raft_consensus.cc:2726] T 2d86f8848a184a87947fd927c44e5a35 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:45.609987 29652 raft_consensus.cc:2454] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 3.
I20240501 20:01:45.610808 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6797d82dda2948d8a7445fcd33e44752" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:45.614641 31212 leader_election.cc:288] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:45.616250 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfac893b4b4f457694a4eb184b6abb53" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:45.617826 30944 raft_consensus.cc:482] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:45.618348 30944 raft_consensus.cc:504] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:45.626215 30944 leader_election.cc:288] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:45.627971 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05df43bdfa479b902b1da297fb5974" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:45.606501 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49da4246f5f6438abe0e7e8c9a523d80" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:45.631289 29806 raft_consensus.cc:2379] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 6: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:45.635474 29657 raft_consensus.cc:1261] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240501 20:01:45.638031 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af08384057a4d85977481472f460168" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:45.631780 31223 leader_election.cc:288] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:45.639859 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9046e38fc57f4364aa3d539e1fc01928" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:45.640870 29732 raft_consensus.cc:2454] T 9046e38fc57f4364aa3d539e1fc01928 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 888d99522e4844118f8f1cf26f268c8c in term 3.
I20240501 20:01:45.639449 29556 catalog_manager.cc:5484] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 1 to 2, leader changed from 4c8da1d580614449a636b855bf11b95f (127.27.217.193) to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 2 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:45.642750 30384 leader_election.cc:302] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 888d99522e4844118f8f1cf26f268c8c, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:45.603078 31267 raft_consensus.cc:482] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting leader election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:45.646941 31267 raft_consensus.cc:3037] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:45.660555 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e89c0572e4ee7be8e5315b459bdb3" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:45.661576 29806 raft_consensus.cc:3037] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:01:45.664846 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48fd0a8e289401aa7417c42ff2c81b2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
W20240501 20:01:45.664458 29773 leader_election.cc:334] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 1.640s (SENT)
I20240501 20:01:45.658608 30944 raft_consensus.cc:2781] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:45.669035 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49da4246f5f6438abe0e7e8c9a523d80" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:45.664083 29556 catalog_manager.cc:5484] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 1 to 4. New cstate: current_term: 4 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:45.673570 29653 raft_consensus.cc:2454] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 5.
I20240501 20:01:45.673916 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77be7101c0fe4085b6706296f609870a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:45.674835 29809 raft_consensus.cc:3037] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:45.671181 30944 raft_consensus.cc:482] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:45.679160 30944 raft_consensus.cc:3037] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [term 3 FOLLOWER]: Advancing to term 4
W20240501 20:01:45.696835 29773 leader_election.cc:334] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 1.691s (SENT)
I20240501 20:01:45.700830 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9046e38fc57f4364aa3d539e1fc01928" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:45.701697 29800 raft_consensus.cc:2454] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 888d99522e4844118f8f1cf26f268c8c in term 3.
I20240501 20:01:45.706642 31197 consensus_queue.cc:1046] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240501 20:01:45.707145 29773 leader_election.cc:302] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:45.715052 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d88139f89c47ff84cad674147aae0f" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:45.713132 29653 raft_consensus.cc:1261] T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240501 20:01:45.714452 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38cbeb0dc0c14e3d9182101eea885625" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:45.717923 29800 raft_consensus.cc:1261] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20240501 20:01:45.767427 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 194237us
I20240501 20:01:45.707814 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05df43bdfa479b902b1da297fb5974" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:45.741935 29556 catalog_manager.cc:5484] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 2 to 4, leader changed from 4c8da1d580614449a636b855bf11b95f (127.27.217.193) to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 4 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:45.715112 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f261d2a05bf4c838532049601694b4a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:45.752071 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8edcd903224d48f889b46e1b5318e0ec" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:45.773231 29652 raft_consensus.cc:2396] T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d for term 5 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240501 20:01:45.763242 29809 raft_consensus.cc:2454] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 4.
W20240501 20:01:45.774499 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 67446.7us
I20240501 20:01:45.718428 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5525fc4b82a44b0991c3dd6098fde81" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:45.774202 29806 raft_consensus.cc:2454] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 7.
I20240501 20:01:45.776459 29804 raft_consensus.cc:2454] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 3.
I20240501 20:01:45.707000 31307 consensus_queue.cc:1035] T 7a3f29f0eeea4b119984634202e5d9cd P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.072s
I20240501 20:01:45.780581 31135 raft_consensus.cc:2726] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:45.784165 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1a08b455b94ae089396a15c837ddc2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:45.782809 29556 catalog_manager.cc:5484] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 3 to 4, leader changed from 4c8da1d580614449a636b855bf11b95f (127.27.217.193) to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 4 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:45.785456 29651 raft_consensus.cc:1226] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 4. Current term is 5. Ops: []
I20240501 20:01:45.785655 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49da4246f5f6438abe0e7e8c9a523d80" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:45.786250 29660 raft_consensus.cc:3037] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:45.745083 31267 raft_consensus.cc:504] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } }
I20240501 20:01:45.781476 31221 raft_consensus.cc:3032] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [term 3 LEADER]: Stepping down as leader of term 3
W20240501 20:01:45.792850 29773 leader_election.cc:334] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 2.034s (SENT)
I20240501 20:01:45.785656 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "456035a1cb3f4f7da52d0fc1dd1c656c" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:45.796286 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8edcd903224d48f889b46e1b5318e0ec" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:45.802103 29659 raft_consensus.cc:2396] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240501 20:01:45.795181 29556 catalog_manager.cc:5484] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 1 to 4. New cstate: current_term: 4 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:45.803634 31268 raft_consensus.cc:482] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:45.804366 31268 raft_consensus.cc:504] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:45.806455 31268 leader_election.cc:288] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:45.807339 30417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9046e38fc57f4364aa3d539e1fc01928" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c" is_pre_election: true
I20240501 20:01:45.803237 29657 raft_consensus.cc:1203] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Deduplicated request from leader. Original: 2.2->[3.3-3.3] Dedup: 3.3->[]
I20240501 20:01:45.808542 29657 raft_consensus.cc:1226] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 3. Current term is 5. Ops: [3.3-3.3]
I20240501 20:01:45.809058 30385 leader_election.cc:302] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 888d99522e4844118f8f1cf26f268c8c; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:45.782254 31063 consensus_queue.cc:1035] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.018s
I20240501 20:01:45.793939 31221 raft_consensus.cc:729] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:45.821002 29556 catalog_manager.cc:5484] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 3 to 4, leader changed from 4c8da1d580614449a636b855bf11b95f (127.27.217.193) to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 4 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:45.824894 29556 catalog_manager.cc:5484] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 2 to 3, leader changed from ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 3 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
W20240501 20:01:45.844610 29773 leader_election.cc:334] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 1.727s (SENT)
I20240501 20:01:45.857158 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49da4246f5f6438abe0e7e8c9a523d80" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
W20240501 20:01:45.851605 29698 connection.cc:232] RPC call timeout handler was delayed by -2.7823728900000333s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1946s for a response
I20240501 20:01:45.856555 31267 leader_election.cc:288] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 election: Requested vote from peers ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:45.863597 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af08384057a4d85977481472f460168" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:45.856374 31383 raft_consensus.cc:2726] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240501 20:01:45.860386 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac2171683884f6f97cdfeb0a656cb49" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:45.856206 31212 raft_consensus.cc:482] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
W20240501 20:01:45.869899 29773 leader_election.cc:334] T 3c5345f6eb62449ba69b4a9c2e349b31 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 1.727s (SENT)
I20240501 20:01:45.870581 29773 leader_election.cc:302] T 3c5345f6eb62449ba69b4a9c2e349b31 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:45.871456 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77be7101c0fe4085b6706296f609870a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:45.872010 29731 raft_consensus.cc:3037] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:45.872732 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5525fc4b82a44b0991c3dd6098fde81" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:45.843948 31221 consensus_queue.cc:260] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:45.870785 29727 raft_consensus.cc:2454] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 4.
W20240501 20:01:45.876693 29698 connection.cc:232] RPC call timeout handler was delayed by -2.3640718190001735s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1572s for a response
W20240501 20:01:45.876942 29698 connection.cc:232] RPC call timeout handler was delayed by -2.251571819000219s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16970000000000002s for a response
W20240501 20:01:45.877111 29698 connection.cc:232] RPC call timeout handler was delayed by -2.0103718190002837s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1965s for a response
I20240501 20:01:45.878616 31230 raft_consensus.cc:482] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:45.881729 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8edcd903224d48f889b46e1b5318e0ec" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:45.882086 31081 raft_consensus.cc:482] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:45.882031 31230 raft_consensus.cc:504] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:45.882509 29726 raft_consensus.cc:2454] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 4.
I20240501 20:01:45.882802 31081 raft_consensus.cc:504] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:45.878700 31367 raft_consensus.cc:482] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:45.878836 31312 raft_consensus.cc:2726] T 3c5345f6eb62449ba69b4a9c2e349b31 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:01:45.885607 31081 leader_election.cc:288] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:45.865891 30944 raft_consensus.cc:504] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:45.888032 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec073fe645614fe28e7837de3c22510b" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:45.883849 31367 raft_consensus.cc:504] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:45.873412 31212 raft_consensus.cc:504] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:45.884632 31230 leader_election.cc:288] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:45.893291 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e07220db7444ae97f294d70eda6a93" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:45.894970 31212 leader_election.cc:288] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:45.895053 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1a08b455b94ae089396a15c837ddc2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:45.895817 31258 raft_consensus.cc:482] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:45.897192 29735 raft_consensus.cc:2396] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240501 20:01:45.897101 31258 raft_consensus.cc:504] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:45.898237 31367 leader_election.cc:288] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:45.898875 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd5e83faaff4e2da4e4cc22ba410d33" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:45.899147 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e9f75387564275a60d38e9a910319c" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:45.899693 29734 raft_consensus.cc:1226] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 2. Current term is 3. Ops: []
I20240501 20:01:45.900244 29808 raft_consensus.cc:1261] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20240501 20:01:45.899195 31258 leader_election.cc:288] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:45.904675 30417 raft_consensus.cc:2379] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 4: Already voted for candidate 888d99522e4844118f8f1cf26f268c8c in this term.
I20240501 20:01:45.916363 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8cd40cb4044d18a86001ce6ddf97ab" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:45.918881 31367 consensus_queue.cc:1035] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:45.920446 31267 raft_consensus.cc:482] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:45.920753 31221 raft_consensus.cc:3037] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:45.920953 31267 raft_consensus.cc:504] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:45.921700 31333 raft_consensus.cc:482] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:45.922214 31333 raft_consensus.cc:504] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:45.922850 31258 raft_consensus.cc:482] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:45.923447 31258 raft_consensus.cc:504] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:45.924805 29655 raft_consensus.cc:1261] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240501 20:01:45.910694 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c651044a305c4168805825a53d910899" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:45.895787 29625 leader_election.cc:302] T 3c5345f6eb62449ba69b4a9c2e349b31 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:45.925374 29734 raft_consensus.cc:2454] T c651044a305c4168805825a53d910899 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 2.
I20240501 20:01:45.921276 31063 raft_consensus.cc:482] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:45.922983 31267 leader_election.cc:288] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:45.929562 29653 raft_consensus.cc:1261] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20240501 20:01:45.933735 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 65882.9us
I20240501 20:01:45.934818 31063 raft_consensus.cc:504] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:45.937068 31063 leader_election.cc:288] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:45.937677 31333 leader_election.cc:288] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:45.927209 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a2bba62f434d1c8489b55c8fec2187" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:45.944904 29625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 98694.2us
W20240501 20:01:45.948264 29772 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.196:37227, user_credentials={real_user=slave}} blocked reactor thread for 118246us
I20240501 20:01:45.948628 29731 raft_consensus.cc:2454] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 4.
I20240501 20:01:45.954684 31153 raft_consensus.cc:2726] T 3c5345f6eb62449ba69b4a9c2e349b31 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:45.939414 31258 leader_election.cc:288] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:45.956930 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9046e38fc57f4364aa3d539e1fc01928" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:45.957615 29731 raft_consensus.cc:3037] T 9046e38fc57f4364aa3d539e1fc01928 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Advancing to term 4
W20240501 20:01:45.958482 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 71712.7us
I20240501 20:01:45.961359 29653 raft_consensus.cc:1261] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240501 20:01:45.909953 29660 raft_consensus.cc:2454] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 6.
I20240501 20:01:45.955488 30944 leader_election.cc:288] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 4 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:45.963765 29773 leader_election.cc:334] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 1.884s (SENT)
I20240501 20:01:45.968003 29773 leader_election.cc:302] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:45.974243 31132 tablet_bootstrap.cc:492] T be383be783234855878cb8fd442c7b05 P 888d99522e4844118f8f1cf26f268c8c: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:01:45.975651 31132 tablet_bootstrap.cc:492] T be383be783234855878cb8fd442c7b05 P 888d99522e4844118f8f1cf26f268c8c: Bootstrap complete.
I20240501 20:01:45.976845 31132 ts_tablet_manager.cc:1367] T be383be783234855878cb8fd442c7b05 P 888d99522e4844118f8f1cf26f268c8c: Time spent bootstrapping tablet: real 0.680s user 0.066s sys 0.056s
I20240501 20:01:45.985613 31153 consensus_queue.cc:1035] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 20:01:45.986418 29698 leader_election.cc:302] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:45.987921 29625 leader_election.cc:302] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters: 888d99522e4844118f8f1cf26f268c8c
I20240501 20:01:45.988673 31230 raft_consensus.cc:2726] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:45.988559 29698 leader_election.cc:302] T c651044a305c4168805825a53d910899 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:45.989869 31230 raft_consensus.cc:2726] T c651044a305c4168805825a53d910899 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:01:45.993302 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77be7101c0fe4085b6706296f609870a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:45.994307 29660 raft_consensus.cc:2354] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d for earlier term 3. Current term is 4.
I20240501 20:01:45.996811 29660 raft_consensus.cc:1226] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 4. Current term is 5. Ops: []
I20240501 20:01:46.001050 29731 raft_consensus.cc:2454] T 9046e38fc57f4364aa3d539e1fc01928 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 888d99522e4844118f8f1cf26f268c8c in term 4.
I20240501 20:01:46.002445 31221 raft_consensus.cc:2726] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:46.002795 30384 leader_election.cc:302] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 888d99522e4844118f8f1cf26f268c8c, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:46.003618 29651 raft_consensus.cc:1261] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240501 20:01:46.003906 31292 raft_consensus.cc:2781] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:01:46.004940 31292 raft_consensus.cc:686] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [term 4 LEADER]: Becoming Leader. State: Replica: 888d99522e4844118f8f1cf26f268c8c, State: Running, Role: LEADER
W20240501 20:01:46.004966 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 45828.5us
I20240501 20:01:46.005942 31292 consensus_queue.cc:237] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:46.006497 31221 consensus_queue.cc:1035] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 20:01:46.008641 29773 leader_election.cc:302] T 48630ae6eb394ac1b54955dd01cf0ed2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
W20240501 20:01:46.010324 29772 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 51265.3us
I20240501 20:01:46.009588 31221 raft_consensus.cc:2726] T 48630ae6eb394ac1b54955dd01cf0ed2 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:01:46.011740 31153 raft_consensus.cc:2781] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 7 FOLLOWER]: Leader election won for term 7
I20240501 20:01:46.011941 31230 consensus_queue.cc:1046] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:46.012740 31153 raft_consensus.cc:686] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [term 7 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:46.012440 31132 raft_consensus.cc:348] T be383be783234855878cb8fd442c7b05 P 888d99522e4844118f8f1cf26f268c8c [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:46.013384 31132 raft_consensus.cc:729] T be383be783234855878cb8fd442c7b05 P 888d99522e4844118f8f1cf26f268c8c [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 888d99522e4844118f8f1cf26f268c8c, State: Initialized, Role: LEARNER
I20240501 20:01:46.013007 31230 raft_consensus.cc:3032] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Stepping down as leader of term 4
I20240501 20:01:46.013979 31230 raft_consensus.cc:729] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:46.014075 31132 consensus_queue.cc:260] T be383be783234855878cb8fd442c7b05 P 888d99522e4844118f8f1cf26f268c8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:46.014156 31153 consensus_queue.cc:237] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:46.016700 31132 ts_tablet_manager.cc:1398] T be383be783234855878cb8fd442c7b05 P 888d99522e4844118f8f1cf26f268c8c: Time spent starting tablet: real 0.039s user 0.007s sys 0.000s
I20240501 20:01:46.027165 29698 leader_election.cc:302] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:46.034071 31367 raft_consensus.cc:2726] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240501 20:01:46.034839 29745 tablet_copy_service.cc:342] P fdfeac4190e045f290e003b2298d988d: Request end of tablet copy session 888d99522e4844118f8f1cf26f268c8c-be383be783234855878cb8fd442c7b05 received from {username='slave'} at 127.0.0.1:47414
I20240501 20:01:46.035379 29745 tablet_copy_service.cc:433] P fdfeac4190e045f290e003b2298d988d: ending tablet copy session 888d99522e4844118f8f1cf26f268c8c-be383be783234855878cb8fd442c7b05 on tablet be383be783234855878cb8fd442c7b05 with peer 888d99522e4844118f8f1cf26f268c8c
I20240501 20:01:46.034313 31230 consensus_queue.cc:260] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:46.033560 29625 leader_election.cc:302] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:46.041608 29656 raft_consensus.cc:1261] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240501 20:01:46.046726 29625 leader_election.cc:302] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:46.047600 31268 raft_consensus.cc:2781] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:01:46.047757 31230 raft_consensus.cc:3037] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:46.048205 31268 raft_consensus.cc:686] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 4 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:46.049134 31268 consensus_queue.cc:237] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:46.055465 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 46890.6us
I20240501 20:01:46.057104 31258 raft_consensus.cc:2781] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240501 20:01:46.057511 31258 raft_consensus.cc:482] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:46.057874 31258 raft_consensus.cc:3037] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:46.055706 29698 leader_election.cc:302] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:46.086980 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5525f546426c47a38edf7f56ad04b7ab" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:46.087157 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0374bb790ed345ed90452595f71383a2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:46.088572 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8cd40cb4044d18a86001ce6ddf97ab" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:46.091336 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece8d4ed84c34b748f7b6272b31c90ac" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:46.096441 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfac893b4b4f457694a4eb184b6abb53" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:46.098341 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be383be783234855878cb8fd442c7b05" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:46.098114 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c651044a305c4168805825a53d910899" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:46.099174 29800 raft_consensus.cc:2454] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 4.
I20240501 20:01:46.067188 31258 raft_consensus.cc:504] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:46.102461 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69915dd5992641e3a24f7f45f1cd5c0c" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:46.090422 31212 raft_consensus.cc:2726] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 20:01:46.110235 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcf30cf66714efdac997862dafbe4a2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:46.112434 29625 leader_election.cc:302] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:46.108286 29802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:46.114475 29698 leader_election.cc:302] T 010ceabf3e5a4b1c9330b80dade765bb P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:46.116991 31258 leader_election.cc:288] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:46.117882 29659 raft_consensus.cc:1261] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240501 20:01:46.118723 29653 raft_consensus.cc:1261] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240501 20:01:46.120362 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84fa28fb55f44b61ba1a88ed775a78ba" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:46.120695 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:46.123147 29726 raft_consensus.cc:2454] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 5.
I20240501 20:01:46.123576 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be383be783234855878cb8fd442c7b05" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:46.124366 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfac893b4b4f457694a4eb184b6abb53" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:46.125186 29734 raft_consensus.cc:2396] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20240501 20:01:46.125874 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69915dd5992641e3a24f7f45f1cd5c0c" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:46.126011 29624 leader_election.cc:302] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:46.126426 29732 raft_consensus.cc:3037] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:46.127108 29624 leader_election.cc:302] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
W20240501 20:01:46.120939 29773 leader_election.cc:334] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 2.186s (SENT)
I20240501 20:01:46.121239 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5525f546426c47a38edf7f56ad04b7ab" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:46.121351 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0374bb790ed345ed90452595f71383a2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:46.131830 29735 raft_consensus.cc:2454] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 4.
I20240501 20:01:46.131985 29730 raft_consensus.cc:2396] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f for term 6 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240501 20:01:46.140750 29733 raft_consensus.cc:2379] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 4: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:46.141947 31267 raft_consensus.cc:2726] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240501 20:01:46.143260 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211e27a3b0084b89a6dd7550849b5a65" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:46.145588 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8edcd903224d48f889b46e1b5318e0ec" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:46.146286 29726 raft_consensus.cc:3037] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:46.150972 31063 raft_consensus.cc:2781] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:46.155365 31063 raft_consensus.cc:482] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting leader election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:46.155732 31063 raft_consensus.cc:3037] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:46.146968 31212 raft_consensus.cc:2726] T 010ceabf3e5a4b1c9330b80dade765bb P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:01:46.149896 31087 raft_consensus.cc:482] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:46.156878 31087 raft_consensus.cc:504] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:46.157685 29698 leader_election.cc:302] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:46.158723 31087 leader_election.cc:288] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:46.159384 31197 consensus_queue.cc:1035] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:46.155107 31135 consensus_queue.cc:1035] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:46.148654 31258 raft_consensus.cc:2726] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240501 20:01:46.164817 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 43895.4us
I20240501 20:01:46.165439 29698 leader_election.cc:302] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:46.167131 31212 raft_consensus.cc:2726] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 20:01:46.167136 31312 consensus_queue.cc:1035] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 20:01:46.169966 29624 leader_election.cc:302] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:46.160142 29802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5525f546426c47a38edf7f56ad04b7ab" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:46.173545 29773 leader_election.cc:334] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 1.944s (SENT)
I20240501 20:01:46.173331 31212 raft_consensus.cc:2726] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:46.190155 29726 raft_consensus.cc:2454] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 5.
I20240501 20:01:46.176205 31258 raft_consensus.cc:2726] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240501 20:01:46.172876 31087 consensus_queue.cc:1046] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.014s
W20240501 20:01:46.208705 29773 leader_election.cc:334] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 1.980s (SENT)
I20240501 20:01:46.213016 29732 raft_consensus.cc:2454] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 5.
I20240501 20:01:46.216334 29698 leader_election.cc:302] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
W20240501 20:01:46.217165 29773 leader_election.cc:334] T 1ca3948261374cef8b1b206a43c00173 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 2.010s (SENT)
I20240501 20:01:46.185249 31063 raft_consensus.cc:504] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:46.218192 29773 leader_election.cc:302] T 1ca3948261374cef8b1b206a43c00173 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:46.219058 31135 raft_consensus.cc:2726] T 1ca3948261374cef8b1b206a43c00173 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:46.219250 31063 leader_election.cc:288] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:46.219652 29773 leader_election.cc:334] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 2.226s (SENT)
I20240501 20:01:46.221614 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9046e38fc57f4364aa3d539e1fc01928" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:46.222160 29803 raft_consensus.cc:3037] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:46.230527 29803 raft_consensus.cc:2454] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 888d99522e4844118f8f1cf26f268c8c in term 4.
I20240501 20:01:46.231070 29772 leader_election.cc:398] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 election: Vote denied by peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959) with higher term. Message: Invalid argument: T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d for earlier term 3. Current term is 4.
I20240501 20:01:46.231426 29772 leader_election.cc:401] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20240501 20:01:46.232034 31221 raft_consensus.cc:2726] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959) with higher term. Message: Invalid argument: T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d for earlier term 3. Current term is 4.
W20240501 20:01:46.233283 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 68316.6us
I20240501 20:01:46.234614 31212 raft_consensus.cc:2781] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:01:46.235180 31212 raft_consensus.cc:686] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:46.236729 29624 leader_election.cc:302] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:46.235958 31212 consensus_queue.cc:237] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:46.240731 31063 raft_consensus.cc:2781] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240501 20:01:46.241088 31063 raft_consensus.cc:482] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting leader election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:46.241429 31063 raft_consensus.cc:3037] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:46.242282 31367 raft_consensus.cc:3032] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:01:46.242671 31367 raft_consensus.cc:729] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:46.242758 31135 consensus_queue.cc:1046] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:01:46.243413 31367 consensus_queue.cc:260] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:46.243678 31307 consensus_queue.cc:1035] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 20:01:46.244848 31367 raft_consensus.cc:3037] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:46.243742 31135 raft_consensus.cc:3032] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [term 4 LEADER]: Stepping down as leader of term 4
I20240501 20:01:46.245254 31135 raft_consensus.cc:729] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:46.246055 31135 consensus_queue.cc:260] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } }
I20240501 20:01:46.247128 31135 raft_consensus.cc:3037] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:46.249258 29698 leader_election.cc:302] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:46.250138 31333 raft_consensus.cc:979] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f: : Instructing follower fdfeac4190e045f290e003b2298d988d to start an election
I20240501 20:01:46.250070 31230 raft_consensus.cc:2726] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:46.250908 31279 raft_consensus.cc:1067] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Signalling peer fdfeac4190e045f290e003b2298d988d to start an election
I20240501 20:01:46.251001 29698 leader_election.cc:302] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:46.251861 31367 raft_consensus.cc:2781] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:46.252118 29698 leader_election.cc:302] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:46.252280 31367 raft_consensus.cc:482] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:46.252926 31367 raft_consensus.cc:3037] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:46.253052 31230 raft_consensus.cc:2726] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240501 20:01:46.252899 29624 leader_election.cc:302] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: 888d99522e4844118f8f1cf26f268c8c, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:46.254505 31258 raft_consensus.cc:2726] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 20:01:46.255869 29624 leader_election.cc:302] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:46.256601 31258 raft_consensus.cc:2726] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240501 20:01:46.257198 29698 leader_election.cc:302] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:46.258550 31230 raft_consensus.cc:2726] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:46.259923 31367 raft_consensus.cc:504] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:46.260201 31063 raft_consensus.cc:504] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:46.259737 29624 leader_election.cc:302] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: 888d99522e4844118f8f1cf26f268c8c, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:46.262527 31258 raft_consensus.cc:2726] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240501 20:01:46.262629 31205 consensus_queue.cc:1035] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:46.263031 29698 leader_election.cc:302] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:46.263751 31063 leader_election.cc:288] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:46.265936 29624 leader_election.cc:302] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:46.266342 31063 consensus_queue.cc:1046] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 20:01:46.267166 31411 consensus_queue.cc:1035] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 20:01:46.268579 29554 catalog_manager.cc:5484] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:46.270117 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9472c53f79ec423f82c396bd980674e2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:46.272140 31367 leader_election.cc:288] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:46.272245 31230 raft_consensus.cc:2726] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240501 20:01:46.273295 31258 raft_consensus.cc:2781] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:46.273824 31258 raft_consensus.cc:482] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting leader election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:46.274292 31258 raft_consensus.cc:3037] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:46.274569 31087 consensus_queue.cc:1035] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:46.274773 29698 leader_election.cc:302] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:46.250072 29556 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:43384 (request call id 144) took 10520 ms. Trace:
I20240501 20:01:46.280275 31205 raft_consensus.cc:2726] T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240501 20:01:46.280612 31333 raft_consensus.cc:3032] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:01:46.280997 31333 raft_consensus.cc:729] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:46.281821 31333 consensus_queue.cc:260] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:46.282944 31333 raft_consensus.cc:3037] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:46.285668 31258 raft_consensus.cc:504] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:46.288141 29624 leader_election.cc:302] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:46.290717 31063 raft_consensus.cc:2781] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240501 20:01:46.291188 31063 raft_consensus.cc:482] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240501 20:01:46.291776 29773 leader_election.cc:334] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 1.801s (SENT)
I20240501 20:01:46.292403 31063 raft_consensus.cc:3037] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:46.292421 29773 leader_election.cc:302] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:46.294874 31258 leader_election.cc:288] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:46.295370 31221 raft_consensus.cc:2726] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240501 20:01:46.291218 29624 leader_election.cc:302] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:46.296636 29698 leader_election.cc:302] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:46.296728 29624 leader_election.cc:302] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:46.300805 31212 raft_consensus.cc:2726] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:46.301113 29624 leader_election.cc:302] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:46.302712 31268 raft_consensus.cc:2726] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240501 20:01:46.302824 31413 raft_consensus.cc:2781] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240501 20:01:46.306396 31413 raft_consensus.cc:482] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:46.306799 31413 raft_consensus.cc:3037] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:46.307343 31268 raft_consensus.cc:2726] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240501 20:01:46.309792 29698 leader_election.cc:302] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:46.310920 31267 raft_consensus.cc:979] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f: : Instructing follower fdfeac4190e045f290e003b2298d988d to start an election
I20240501 20:01:46.311151 31205 raft_consensus.cc:2726] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:46.311379 31267 raft_consensus.cc:1067] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Signalling peer fdfeac4190e045f290e003b2298d988d to start an election
I20240501 20:01:46.311789 31205 consensus_queue.cc:1046] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 20:01:46.312911 31087 raft_consensus.cc:3032] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Stepping down as leader of term 4
I20240501 20:01:46.313254 31087 raft_consensus.cc:729] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:46.314147 31087 consensus_queue.cc:260] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:46.316403 29624 leader_election.cc:302] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:46.317046 31087 raft_consensus.cc:3037] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:46.317166 31153 raft_consensus.cc:2781] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:01:46.317924 31153 raft_consensus.cc:686] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [term 5 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:46.318722 31153 consensus_queue.cc:237] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:46.321560 29773 leader_election.cc:334] T 36057b3f24d44b9aac3a1378053a395f P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 2.102s (SENT)
I20240501 20:01:46.322103 29773 leader_election.cc:302] T 36057b3f24d44b9aac3a1378053a395f P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:46.323385 31312 raft_consensus.cc:2726] T 36057b3f24d44b9aac3a1378053a395f P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:46.324236 29698 leader_election.cc:302] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:46.326224 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be383be783234855878cb8fd442c7b05" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:46.327222 29698 leader_election.cc:302] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:46.304468 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8edcd903224d48f889b46e1b5318e0ec" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:46.328342 29805 raft_consensus.cc:3037] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Advancing to term 5
I20240501 20:01:46.331265 31413 raft_consensus.cc:504] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:46.278071 29556 rpcz_store.cc:276] 0501 20:01:35.729543 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:35.748218 (+ 18675us) service_pool.cc:225] Handling call
0501 20:01:46.249983 (+10501765us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0501 20:01:46.030224 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 20:01:46.030709 (+ 485us) write_op.cc:424] Acquiring schema lock in shared mode
0501 20:01:46.030765 (+ 56us) write_op.cc:427] Acquired schema lock
0501 20:01:46.030781 (+ 16us) tablet.cc:660] Decoding operations
0501 20:01:46.031853 (+ 1072us) write_op.cc:612] Acquiring the partition lock for write op
0501 20:01:46.031927 (+ 74us) write_op.cc:633] Partition lock acquired for write op
0501 20:01:46.031942 (+ 15us) tablet.cc:683] Acquiring locks for 14 operations
0501 20:01:46.048495 (+ 16553us) tablet.cc:699] Row locks acquired
0501 20:01:46.048513 (+ 18us) write_op.cc:260] PREPARE: finished
0501 20:01:46.048664 (+ 151us) write_op.cc:270] Start()
0501 20:01:46.048813 (+ 149us) write_op.cc:276] Timestamp: P: 1714593706048625 usec, L: 0
0501 20:01:46.048843 (+ 30us) op_driver.cc:331] REPLICATION: starting
0501 20:01:46.049471 (+ 628us) log.cc:842] Serialized 5618 byte log entry
0501 20:01:46.153010 (+103539us) op_driver.cc:444] REPLICATION: finished
0501 20:01:46.163246 (+ 10236us) write_op.cc:301] APPLY: starting
0501 20:01:46.163330 (+ 84us) tablet.cc:1365] starting BulkCheckPresence
0501 20:01:46.163606 (+ 276us) tablet.cc:1368] finished BulkCheckPresence
0501 20:01:46.163814 (+ 208us) tablet.cc:1370] starting ApplyRowOperation cycle
0501 20:01:46.208156 (+ 44342us) tablet.cc:1381] finished ApplyRowOperation cycle
0501 20:01:46.208441 (+ 285us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=14
0501 20:01:46.208507 (+ 66us) write_op.cc:312] APPLY: finished
0501 20:01:46.208880 (+ 373us) log.cc:842] Serialized 100 byte log entry
0501 20:01:46.209591 (+ 711us) write_op.cc:481] Releasing partition, row and schema locks
0501 20:01:46.209735 (+ 144us) write_op.cc:446] Released schema lock
0501 20:01:46.209792 (+ 57us) write_op.cc:341] FINISH: Updating metrics
0501 20:01:46.210040 (+ 248us) write_op.cc:481] Releasing partition, row and schema locks
0501 20:01:46.210064 (+ 24us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":4460,"reported_tablets":32,"child_traces":[["op",{"apply.queue_time_us":7169,"apply.run_cpu_time_us":4508,"apply.run_wall_time_us":48372,"num_ops":14,"prepare.queue_time_us":63821,"prepare.run_cpu_time_us":11651,"prepare.run_wall_time_us":70812,"raft.queue_time_us":9222,"raft.run_cpu_time_us":1362,"raft.run_wall_time_us":1440,"replication_time_us":104047,"thread_start_us":97747,"threads_started":4,"wal-append.queue_time_us":66451}]]}
I20240501 20:01:46.336251 29805 raft_consensus.cc:2454] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 5.
I20240501 20:01:46.337616 29803 raft_consensus.cc:2454] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 4.
I20240501 20:01:46.340456 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac2171683884f6f97cdfeb0a656cb49" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:46.341066 29802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c651044a305c4168805825a53d910899" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:46.344262 31205 raft_consensus.cc:2726] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240501 20:01:46.327756 31087 raft_consensus.cc:2726] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:46.317229 29624 leader_election.cc:302] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:46.350936 31268 raft_consensus.cc:2781] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:01:46.351482 31268 raft_consensus.cc:686] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [term 5 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:46.350872 29557 catalog_manager.cc:5484] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c reported cstate change: term changed from 1 to 4, leader changed from ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) to 888d99522e4844118f8f1cf26f268c8c (127.27.217.196). New cstate: current_term: 4 leader_uuid: "888d99522e4844118f8f1cf26f268c8c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:01:46.352356 31268 consensus_queue.cc:237] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } }
W20240501 20:01:46.354876 29773 leader_election.cc:334] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): Timed out: RequestConsensusVote RPC to 127.27.217.194:32907 timed out after 2.422s (SENT)
I20240501 20:01:46.362643 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be383be783234855878cb8fd442c7b05" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:46.363363 29732 raft_consensus.cc:2379] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 4: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:46.364917 31087 consensus_queue.cc:1046] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:01:46.366166 31087 raft_consensus.cc:3032] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:01:46.366484 31087 raft_consensus.cc:729] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:46.383249 30417 raft_consensus.cc:1261] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [term 5 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240501 20:01:46.384666 29731 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aa8cd40cb4044d18a86001ce6ddf97ab"
dest_uuid: "fdfeac4190e045f290e003b2298d988d"
from {username='slave'} at 127.0.0.1:36270
I20240501 20:01:46.385303 29731 raft_consensus.cc:482] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:01:46.385710 29731 raft_consensus.cc:3037] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:46.389561 29773 leader_election.cc:302] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:46.393244 29625 leader_election.cc:302] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:46.402343 29653 raft_consensus.cc:1226] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20240501 20:01:46.371294 31307 consensus_queue.cc:1035] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.012s
I20240501 20:01:46.403553 29558 catalog_manager.cc:5484] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 2 to 3, leader changed from fdfeac4190e045f290e003b2298d988d (127.27.217.194) to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 3 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } }
I20240501 20:01:46.397836 29731 raft_consensus.cc:504] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:46.411438 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac2171683884f6f97cdfeb0a656cb49" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:46.412014 29732 raft_consensus.cc:3037] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:46.413002 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c651044a305c4168805825a53d910899" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:46.413547 29733 raft_consensus.cc:3037] T c651044a305c4168805825a53d910899 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Advancing to term 4
I20240501 20:01:46.413661 29731 leader_election.cc:288] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:46.370988 31087 consensus_queue.cc:260] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:46.424824 29558 catalog_manager.cc:5484] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } }
I20240501 20:01:46.427634 29558 catalog_manager.cc:5484] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 2 to 3, leader changed from fdfeac4190e045f290e003b2298d988d (127.27.217.194) to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 3 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
I20240501 20:01:46.429630 29558 catalog_manager.cc:5484] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 1 to 3, leader changed from ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 3 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:46.435887 29731 raft_consensus.cc:1261] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20240501 20:01:46.437705 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 48231.2us
I20240501 20:01:46.438400 29732 raft_consensus.cc:2454] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 5.
I20240501 20:01:46.440178 31268 raft_consensus.cc:2781] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:01:46.441072 29558 catalog_manager.cc:5484] T 456035a1cb3f4f7da52d0fc1dd1c656c P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 2 to 3, leader changed from fdfeac4190e045f290e003b2298d988d (127.27.217.194) to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 3 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:46.443212 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649b71f9f267437ca51c758dd9ca6987" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:46.372556 31063 raft_consensus.cc:504] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:46.439111 29554 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:43396 (request call id 162) took 10463 ms. Trace:
I20240501 20:01:46.419525 31197 raft_consensus.cc:3037] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:46.422360 29729 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9f261d2a05bf4c838532049601694b4a"
dest_uuid: "fdfeac4190e045f290e003b2298d988d"
from {username='slave'} at 127.0.0.1:36270
I20240501 20:01:46.443857 29558 catalog_manager.cc:5484] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } }
I20240501 20:01:46.450600 29558 catalog_manager.cc:5484] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } }
I20240501 20:01:46.452593 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8cd40cb4044d18a86001ce6ddf97ab" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:46.438107 29802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8cd40cb4044d18a86001ce6ddf97ab" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:46.453903 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece8d4ed84c34b748f7b6272b31c90ac" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:46.454067 29802 raft_consensus.cc:3037] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:46.454576 29726 raft_consensus.cc:2454] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 5.
I20240501 20:01:46.419694 31312 consensus_queue.cc:1035] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
I20240501 20:01:46.461434 29656 raft_consensus.cc:1226] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20240501 20:01:46.452643 31413 leader_election.cc:288] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:46.464218 29729 raft_consensus.cc:482] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:01:46.464651 29729 raft_consensus.cc:3037] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:46.465806 31279 raft_consensus.cc:979] T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f: : Instructing follower fdfeac4190e045f290e003b2298d988d to start an election
I20240501 20:01:46.446040 29554 rpcz_store.cc:276] 0501 20:01:35.966976 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:35.967346 (+ 370us) service_pool.cc:225] Handling call
0501 20:01:46.315653 (+10348307us) write_op.cc:481] Releasing partition, row and schema locks
0501 20:01:46.315680 (+ 27us) write_op.cc:446] Released schema lock
0501 20:01:46.430305 (+114625us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0501 20:01:46.295600 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 20:01:46.296097 (+ 497us) write_op.cc:424] Acquiring schema lock in shared mode
0501 20:01:46.296131 (+ 34us) write_op.cc:427] Acquired schema lock
0501 20:01:46.296143 (+ 12us) tablet.cc:660] Decoding operations
0501 20:01:46.296468 (+ 325us) write_op.cc:612] Acquiring the partition lock for write op
0501 20:01:46.296551 (+ 83us) write_op.cc:633] Partition lock acquired for write op
0501 20:01:46.296580 (+ 29us) tablet.cc:683] Acquiring locks for 1 operations
0501 20:01:46.296758 (+ 178us) tablet.cc:699] Row locks acquired
0501 20:01:46.296779 (+ 21us) write_op.cc:260] PREPARE: finished
0501 20:01:46.296907 (+ 128us) write_op.cc:270] Start()
0501 20:01:46.296997 (+ 90us) write_op.cc:276] Timestamp: P: 1714593706296867 usec, L: 0
0501 20:01:46.297017 (+ 20us) op_driver.cc:331] REPLICATION: starting
0501 20:01:46.297519 (+ 502us) log.cc:842] Serialized 543 byte log entry
0501 20:01:46.299773 (+ 2254us) op_driver.cc:444] REPLICATION: finished
0501 20:01:46.300253 (+ 480us) write_op.cc:301] APPLY: starting
0501 20:01:46.300340 (+ 87us) tablet.cc:1365] starting BulkCheckPresence
0501 20:01:46.300471 (+ 131us) tablet.cc:1368] finished BulkCheckPresence
0501 20:01:46.300599 (+ 128us) tablet.cc:1370] starting ApplyRowOperation cycle
0501 20:01:46.300817 (+ 218us) tablet.cc:1381] finished ApplyRowOperation cycle
0501 20:01:46.300953 (+ 136us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0501 20:01:46.300992 (+ 39us) write_op.cc:312] APPLY: finished
0501 20:01:46.301149 (+ 157us) log.cc:842] Serialized 22 byte log entry
0501 20:01:46.301298 (+ 149us) write_op.cc:481] Releasing partition, row and schema locks
0501 20:01:46.301370 (+ 72us) write_op.cc:446] Released schema lock
0501 20:01:46.301406 (+ 36us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":32605,"reported_tablets":30,"child_traces":[["op",{"apply.queue_time_us":250,"apply.run_cpu_time_us":1424,"apply.run_wall_time_us":1425,"num_ops":1,"prepare.queue_time_us":675,"prepare.run_cpu_time_us":2196,"prepare.run_wall_time_us":2196,"replication_time_us":2654}]]}
I20240501 20:01:46.470911 31087 raft_consensus.cc:3037] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:46.473138 29773 leader_election.cc:302] T a8e9f75387564275a60d38e9a910319c P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:46.475054 29773 leader_election.cc:302] T 010ceabf3e5a4b1c9330b80dade765bb P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:46.476276 31411 raft_consensus.cc:2726] T a8e9f75387564275a60d38e9a910319c P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:01:46.476817 31411 raft_consensus.cc:2726] T 010ceabf3e5a4b1c9330b80dade765bb P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:01:46.478479 29735 raft_consensus.cc:1261] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20240501 20:01:46.478840 29735 pending_rounds.cc:76] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d: Aborting all ops after (but not including) 2
I20240501 20:01:46.479056 29735 pending_rounds.cc:97] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20240501 20:01:46.479254 29735 raft_consensus.cc:2866] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 20:01:46.471031 29730 raft_consensus.cc:1261] T 9046e38fc57f4364aa3d539e1fc01928 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Refusing update from remote peer 888d99522e4844118f8f1cf26f268c8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
W20240501 20:01:46.480867 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 86463.1us
I20240501 20:01:46.484236 29773 leader_election.cc:302] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 888d99522e4844118f8f1cf26f268c8c, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:46.492000 29734 raft_consensus.cc:2454] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 5.
I20240501 20:01:46.480106 29558 catalog_manager.cc:5484] T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } }
I20240501 20:01:46.496390 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9046e38fc57f4364aa3d539e1fc01928" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:46.497180 29732 raft_consensus.cc:2379] T 9046e38fc57f4364aa3d539e1fc01928 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 4: Already voted for candidate 888d99522e4844118f8f1cf26f268c8c in this term.
I20240501 20:01:46.497077 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84fa28fb55f44b61ba1a88ed775a78ba" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:46.495360 29558 catalog_manager.cc:5484] T 010ceabf3e5a4b1c9330b80dade765bb P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 1 to 2, leader changed from ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 2 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:46.499233 29558 catalog_manager.cc:5484] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:46.501941 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211e27a3b0084b89a6dd7550849b5a65" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:46.503270 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8787bc4d6b44649b9138d906730110a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:46.510918 29730 raft_consensus.cc:2379] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 3: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:46.515105 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:46.515656 29732 raft_consensus.cc:3037] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:46.516024 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece8d4ed84c34b748f7b6272b31c90ac" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:46.519511 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0374bb790ed345ed90452595f71383a2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:46.520072 29735 raft_consensus.cc:3037] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:46.521131 29729 raft_consensus.cc:504] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:46.523151 29729 leader_election.cc:288] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:46.525801 29730 raft_consensus.cc:1226] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 4. Current term is 5. Ops: []
W20240501 20:01:46.531808 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 55746us
I20240501 20:01:46.506145 29558 catalog_manager.cc:5484] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
W20240501 20:01:46.536106 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 52680.3us
I20240501 20:01:46.540410 29735 raft_consensus.cc:2454] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 5.
I20240501 20:01:46.542649 29624 leader_election.cc:302] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:46.545717 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f261d2a05bf4c838532049601694b4a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:46.546427 29805 raft_consensus.cc:3037] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:46.550611 29735 raft_consensus.cc:1261] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240501 20:01:46.551561 29558 catalog_manager.cc:5484] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } }
I20240501 20:01:46.514294 31063 leader_election.cc:288] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:46.495975 30944 consensus_queue.cc:1035] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.046s
I20240501 20:01:46.557614 29558 catalog_manager.cc:5484] T f6d88139f89c47ff84cad674147aae0f P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:46.515105 31268 raft_consensus.cc:686] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [term 4 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:46.514720 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcf30cf66714efdac997862dafbe4a2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:46.563664 29734 raft_consensus.cc:1226] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 3. Current term is 4. Ops: []
I20240501 20:01:46.566884 29731 raft_consensus.cc:1261] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240501 20:01:46.535507 31307 raft_consensus.cc:3037] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:46.514885 31221 consensus_queue.cc:1046] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.046s
I20240501 20:01:46.578706 29732 raft_consensus.cc:2454] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 6.
I20240501 20:01:46.571273 29730 raft_consensus.cc:1226] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 4. Current term is 5. Ops: []
I20240501 20:01:46.553508 31043 raft_consensus.cc:482] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:46.535645 31411 consensus_queue.cc:1046] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20240501 20:01:46.570791 29558 catalog_manager.cc:5484] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } }
W20240501 20:01:46.597748 29625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 173274us
I20240501 20:01:46.600786 29733 raft_consensus.cc:2454] T c651044a305c4168805825a53d910899 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 4.
I20240501 20:01:46.599992 29558 catalog_manager.cc:5484] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 1 to 4. New cstate: current_term: 4 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } }
I20240501 20:01:46.602717 29805 raft_consensus.cc:2454] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 4.
I20240501 20:01:46.572717 31063 raft_consensus.cc:1067] T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f [term 2 LEADER]: Signalling peer fdfeac4190e045f290e003b2298d988d to start an election
I20240501 20:01:46.567831 31432 raft_consensus.cc:482] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:46.604190 29700 leader_election.cc:302] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:46.610842 29728 raft_consensus.cc:1226] T c651044a305c4168805825a53d910899 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 3. Current term is 4. Ops: []
W20240501 20:01:46.613899 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 45289us
I20240501 20:01:46.607705 29726 raft_consensus.cc:2454] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 5.
I20240501 20:01:46.608414 31043 raft_consensus.cc:504] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:46.625497 31411 raft_consensus.cc:3032] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 5 LEADER]: Stepping down as leader of term 5
I20240501 20:01:46.626097 31411 raft_consensus.cc:729] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:46.627059 31043 leader_election.cc:288] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:46.628253 31210 consensus_queue.cc:1035] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.086s
I20240501 20:01:46.608845 31333 raft_consensus.cc:2781] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:01:46.634763 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e38a2e91a3d48d09a22abce723af833" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:46.636710 29805 raft_consensus.cc:2396] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d for term 5 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240501 20:01:46.633635 31428 raft_consensus.cc:3032] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 4 LEADER]: Stepping down as leader of term 4
I20240501 20:01:46.627204 31411 consensus_queue.cc:260] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:46.720120 29625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 119083us
I20240501 20:01:46.764683 29726 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a8e9f75387564275a60d38e9a910319c"
dest_uuid: "fdfeac4190e045f290e003b2298d988d"
from {username='slave'} at 127.0.0.1:36270
I20240501 20:01:46.639621 31081 raft_consensus.cc:2781] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:01:46.768779 29726 raft_consensus.cc:482] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:01:46.769393 29726 raft_consensus.cc:3037] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:46.761504 30944 raft_consensus.cc:482] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:46.770344 29773 leader_election.cc:302] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:46.664602 31333 raft_consensus.cc:686] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [term 5 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:46.716185 29729 raft_consensus.cc:1261] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20240501 20:01:46.777727 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 55659.3us
I20240501 20:01:46.763101 31428 raft_consensus.cc:729] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
W20240501 20:01:46.783171 29772 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 120743us
I20240501 20:01:46.632148 31432 raft_consensus.cc:504] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:46.779516 30944 raft_consensus.cc:504] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:46.800822 29726 raft_consensus.cc:504] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:46.781319 29802 raft_consensus.cc:2454] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 2.
I20240501 20:01:46.824858 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:46.828229 29806 raft_consensus.cc:1226] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:46.826856 29726 leader_election.cc:288] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:46.830042 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0374bb790ed345ed90452595f71383a2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:46.805481 31411 raft_consensus.cc:3037] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Advancing to term 6
W20240501 20:01:46.833201 31431 log.cc:924] Time spent T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 0.171s user 0.000s sys 0.001s
I20240501 20:01:46.833949 29658 raft_consensus.cc:1226] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 3. Current term is 4. Ops: []
W20240501 20:01:46.834733 31429 log.cc:924] Time spent T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
W20240501 20:01:46.835538 31430 log.cc:924] Time spent T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20240501 20:01:46.838243 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e9f75387564275a60d38e9a910319c" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
W20240501 20:01:46.839843 31422 log.cc:924] Time spent T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 0.208s user 0.001s sys 0.000s
I20240501 20:01:46.807451 31135 raft_consensus.cc:2726] T 03f5d1686214485d91a0baf4b870740e P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:46.842087 31432 leader_election.cc:288] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:46.846948 31444 raft_consensus.cc:482] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:46.847128 29653 raft_consensus.cc:1261] T 3c5345f6eb62449ba69b4a9c2e349b31 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 20:01:46.848529 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcf30cf66714efdac997862dafbe4a2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:46.856153 29654 raft_consensus.cc:1261] T 1ca3948261374cef8b1b206a43c00173 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:01:46.866338 31444 raft_consensus.cc:504] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:46.861321 29808 raft_consensus.cc:3037] T a8e9f75387564275a60d38e9a910319c P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:46.767921 31268 consensus_queue.cc:237] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:46.868541 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e9f75387564275a60d38e9a910319c" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:46.872112 29700 leader_election.cc:302] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:46.898283 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9472c53f79ec423f82c396bd980674e2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:46.902709 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8cd40cb4044d18a86001ce6ddf97ab" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:46.908468 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8cd40cb4044d18a86001ce6ddf97ab" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:46.909072 29656 raft_consensus.cc:3032] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:46.922592 29656 raft_consensus.cc:729] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:46.886202 31197 raft_consensus.cc:2726] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:46.925350 29800 raft_consensus.cc:3037] T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:46.893218 29654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a2bba62f434d1c8489b55c8fec2187" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
W20240501 20:01:46.927793 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 210398us
I20240501 20:01:46.928220 29654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec073fe645614fe28e7837de3c22510b" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:46.928555 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfac893b4b4f457694a4eb184b6abb53" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:46.929464 29654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e9f75387564275a60d38e9a910319c" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:46.929512 29773 leader_election.cc:302] T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:46.930124 29654 raft_consensus.cc:3032] T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:01:46.930480 29654 raft_consensus.cc:729] T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:46.931207 29773 leader_election.cc:398] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 election: Vote denied by peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907) with higher term. Message: Invalid argument: T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d for earlier term 4. Current term is 5.
I20240501 20:01:46.934049 29773 leader_election.cc:302] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:46.929571 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e38a2e91a3d48d09a22abce723af833" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:46.924369 29656 consensus_queue.cc:260] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:46.899981 31442 raft_consensus.cc:2781] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:01:46.931227 29654 consensus_queue.cc:260] T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:46.943431 29804 raft_consensus.cc:1261] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240501 20:01:46.943527 29654 raft_consensus.cc:3037] T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:46.944607 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5525f546426c47a38edf7f56ad04b7ab" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:46.945214 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd5e83faaff4e2da4e4cc22ba410d33" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:46.945773 29655 raft_consensus.cc:2454] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 888d99522e4844118f8f1cf26f268c8c in term 6.
I20240501 20:01:46.946640 31442 raft_consensus.cc:686] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [term 2 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:46.946851 29655 raft_consensus.cc:1261] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20240501 20:01:46.948004 29653 raft_consensus.cc:2396] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d for term 5 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240501 20:01:46.948318 29698 leader_election.cc:302] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:46.949040 29800 raft_consensus.cc:1261] T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240501 20:01:46.949440 29698 leader_election.cc:302] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:46.938540 29656 raft_consensus.cc:3037] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:46.909385 31212 raft_consensus.cc:482] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 888d99522e4844118f8f1cf26f268c8c)
I20240501 20:01:46.955798 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e38a2e91a3d48d09a22abce723af833" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:46.852584 31333 consensus_queue.cc:237] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:46.960922 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f261d2a05bf4c838532049601694b4a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:46.961756 29809 raft_consensus.cc:2454] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 4.
I20240501 20:01:46.940289 29651 raft_consensus.cc:1261] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240501 20:01:46.968647 29624 leader_election.cc:302] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:46.970752 29624 leader_election.cc:302] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:46.971827 29806 raft_consensus.cc:1261] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 7 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20240501 20:01:46.939581 31087 raft_consensus.cc:482] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 888d99522e4844118f8f1cf26f268c8c)
I20240501 20:01:46.977762 29800 raft_consensus.cc:1261] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240501 20:01:46.983762 29653 raft_consensus.cc:1261] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240501 20:01:46.958577 31197 raft_consensus.cc:2781] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240501 20:01:46.959419 31445 raft_consensus.cc:2726] T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240501 20:01:46.986624 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 54619.7us
W20240501 20:01:46.986595 30384 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 175976us
I20240501 20:01:46.990293 29624 leader_election.cc:302] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:46.991955 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:01:46.969483 30944 leader_election.cc:288] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:46.868860 31428 consensus_queue.cc:260] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:46.995918 29773 leader_election.cc:302] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:46.956980 31210 consensus_queue.cc:1035] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
I20240501 20:01:46.982100 31205 raft_consensus.cc:2726] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:01:47.020303 29625 leader_election.cc:302] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:46.988644 31197 raft_consensus.cc:482] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:46.969944 31212 raft_consensus.cc:504] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:47.025208 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e38a2e91a3d48d09a22abce723af833" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:46.992558 31444 leader_election.cc:288] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:47.046643 29728 raft_consensus.cc:1261] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240501 20:01:47.008864 31439 raft_consensus.cc:482] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:47.045096 29732 raft_consensus.cc:2454] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 4.
I20240501 20:01:47.020076 31221 raft_consensus.cc:2726] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:47.017247 31081 raft_consensus.cc:686] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:47.024025 31428 raft_consensus.cc:3037] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:46.985183 31087 raft_consensus.cc:504] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
W20240501 20:01:47.105542 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 109668us
I20240501 20:01:47.108367 29658 raft_consensus.cc:3032] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:01:47.108929 29800 raft_consensus.cc:1226] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 3. Current term is 4. Ops: []
I20240501 20:01:47.046056 31197 raft_consensus.cc:3037] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:47.023177 31421 raft_consensus.cc:2781] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Leader election won for term 6
I20240501 20:01:47.129711 31433 consensus_queue.cc:1035] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 4, Time since last communication: 0.024s
I20240501 20:01:47.131036 31437 raft_consensus.cc:2781] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:01:47.131512 31437 raft_consensus.cc:686] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f [term 4 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:47.132222 30383 leader_election.cc:302] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, 888d99522e4844118f8f1cf26f268c8c; no voters:
W20240501 20:01:47.132936 30383 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 140647us
I20240501 20:01:47.049993 31258 raft_consensus.cc:2781] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240501 20:01:47.057667 31367 raft_consensus.cc:2726] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240501 20:01:47.119702 29808 raft_consensus.cc:2454] T a8e9f75387564275a60d38e9a910319c P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 3.
I20240501 20:01:47.122188 29658 raft_consensus.cc:729] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:47.163779 29804 raft_consensus.cc:1226] T a8e9f75387564275a60d38e9a910319c P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 2. Current term is 3. Ops: []
I20240501 20:01:47.129038 31453 consensus_queue.cc:1035] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.070s
I20240501 20:01:47.129330 31267 raft_consensus.cc:482] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:47.129923 31421 raft_consensus.cc:686] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 6 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:47.133112 31450 raft_consensus.cc:2781] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240501 20:01:47.132385 31437 consensus_queue.cc:237] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:47.133323 31440 consensus_queue.cc:1035] T 1ca3948261374cef8b1b206a43c00173 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.154s
I20240501 20:01:46.986344 31442 consensus_queue.cc:237] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:47.133317 31427 raft_consensus.cc:2781] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:01:47.177367 31427 raft_consensus.cc:686] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 5 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:47.178210 31427 consensus_queue.cc:237] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:47.179358 29733 raft_consensus.cc:1261] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240501 20:01:47.133769 31425 consensus_queue.cc:1035] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.114s
W20240501 20:01:47.048796 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 97760us
W20240501 20:01:47.118184 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 168843us
W20240501 20:01:47.118955 31412 log.cc:924] Time spent T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20240501 20:01:47.183494 31212 leader_election.cc:288] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:47.183802 29698 leader_election.cc:302] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:47.114089 31426 raft_consensus.cc:482] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:47.185308 31426 raft_consensus.cc:504] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:47.107548 31063 consensus_queue.cc:1035] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.053s
I20240501 20:01:47.043807 31446 consensus_queue.cc:1046] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.141s
I20240501 20:01:47.186724 31446 consensus_queue.cc:1035] T 38cbeb0dc0c14e3d9182101eea885625 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:47.167917 29658 consensus_queue.cc:260] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:47.188068 29658 raft_consensus.cc:3037] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:47.176721 31435 raft_consensus.cc:482] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
W20240501 20:01:47.176929 31441 log.cc:924] Time spent T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.052s user 0.000s sys 0.002s
I20240501 20:01:47.185740 31449 consensus_queue.cc:1035] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:47.164806 31413 raft_consensus.cc:482] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:47.162761 31258 raft_consensus.cc:482] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:47.191354 29656 raft_consensus.cc:2454] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 2.
I20240501 20:01:47.062213 31439 raft_consensus.cc:504] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:47.168848 31432 raft_consensus.cc:482] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:47.172092 31450 raft_consensus.cc:482] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240501 20:01:47.212255 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 65927.6us
I20240501 20:01:47.221973 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e07220db7444ae97f294d70eda6a93" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:47.196583 31258 raft_consensus.cc:3037] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:47.226276 29656 raft_consensus.cc:2454] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 888d99522e4844118f8f1cf26f268c8c in term 4.
I20240501 20:01:47.172221 31267 raft_consensus.cc:504] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:47.241544 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e89c0572e4ee7be8e5315b459bdb3" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:47.242478 29654 raft_consensus.cc:2454] T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 3.
I20240501 20:01:47.213991 31448 raft_consensus.cc:482] T 1ca3948261374cef8b1b206a43c00173 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:47.244971 31448 raft_consensus.cc:504] T 1ca3948261374cef8b1b206a43c00173 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:47.247345 31221 raft_consensus.cc:3032] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:01:47.247808 31221 raft_consensus.cc:729] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:47.213575 31421 consensus_queue.cc:237] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:47.248598 31439 leader_election.cc:288] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:47.248631 31221 consensus_queue.cc:260] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:47.212708 30414 raft_consensus.cc:1261] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 7 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20240501 20:01:47.250646 31221 raft_consensus.cc:3037] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:47.253481 31153 consensus_queue.cc:1035] T fa7e89c0572e4ee7be8e5315b459bdb3 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240501 20:01:47.254105 31087 leader_election.cc:288] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
W20240501 20:01:47.256583 31307 consensus_meta.cc:287] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d: Time spent flushing consensus metadata: real 0.660s user 0.003s sys 0.007s
I20240501 20:01:47.257025 31307 raft_consensus.cc:2726] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:47.259040 29773 leader_election.cc:302] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:47.209955 31413 raft_consensus.cc:504] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:47.195736 31435 raft_consensus.cc:504] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:47.210832 31450 raft_consensus.cc:3037] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [term 6 FOLLOWER]: Advancing to term 7
W20240501 20:01:47.278293 29606 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica 4c8da1d580614449a636b855bf11b95f is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } }
I20240501 20:01:47.216197 31432 raft_consensus.cc:504] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:47.285077 31432 leader_election.cc:288] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:47.290283 29606 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet a8e9f75387564275a60d38e9a910319c, TS fdfeac4190e045f290e003b2298d988d -> TS 888d99522e4844118f8f1cf26f268c8c move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:47.290688 29606 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet a8e9f75387564275a60d38e9a910319c, TS fdfeac4190e045f290e003b2298d988d -> TS 888d99522e4844118f8f1cf26f268c8c move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:47.303438 29805 raft_consensus.cc:2396] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
W20240501 20:01:47.316627 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 94862us
W20240501 20:01:47.317230 29772 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 170611us
I20240501 20:01:47.293310 31432 raft_consensus.cc:482] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:47.215571 31081 consensus_queue.cc:237] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:47.306934 31438 raft_consensus.cc:482] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:47.332109 29658 raft_consensus.cc:2454] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 4.
I20240501 20:01:47.311964 31411 raft_consensus.cc:2781] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:01:47.289122 30383 leader_election.cc:302] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, 888d99522e4844118f8f1cf26f268c8c; no voters:
I20240501 20:01:47.320995 31432 raft_consensus.cc:504] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:47.394554 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 61069.2us
I20240501 20:01:47.355944 31471 raft_consensus.cc:482] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240501 20:01:47.401880 31458 log.cc:924] Time spent T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 0.146s user 0.001s sys 0.000s
I20240501 20:01:47.345722 31435 leader_election.cc:288] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:47.408013 29772 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 88112.9us
I20240501 20:01:47.385350 31411 raft_consensus.cc:686] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [term 4 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
W20240501 20:01:47.387018 30383 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 117811us
I20240501 20:01:47.392872 31439 raft_consensus.cc:2781] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240501 20:01:47.337255 31438 raft_consensus.cc:504] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:47.421335 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 237606us
I20240501 20:01:47.422111 29698 leader_election.cc:302] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:47.427772 31453 raft_consensus.cc:2726] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 20:01:47.434075 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca3948261374cef8b1b206a43c00173" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:47.435180 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e49dd5180f04313ab33942fb3ddc147" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:47.422693 31439 raft_consensus.cc:482] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:47.435884 31439 raft_consensus.cc:3037] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:47.435276 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31fff64b453a423eb0f567b3171564fb" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:47.436019 29807 raft_consensus.cc:2379] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 5: Already voted for candidate ee0d97247d394a8facfa774f15f3a67d in this term.
I20240501 20:01:47.437821 29773 leader_election.cc:302] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:47.436544 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece8d4ed84c34b748f7b6272b31c90ac" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:47.439942 29809 raft_consensus.cc:2379] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 5: Already voted for candidate ee0d97247d394a8facfa774f15f3a67d in this term.
I20240501 20:01:47.442283 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31fff64b453a423eb0f567b3171564fb" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:47.448180 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca3948261374cef8b1b206a43c00173" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:47.470088 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece8d4ed84c34b748f7b6272b31c90ac" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:47.471381 29731 raft_consensus.cc:2379] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 5: Already voted for candidate ee0d97247d394a8facfa774f15f3a67d in this term.
I20240501 20:01:47.471735 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd5e83faaff4e2da4e4cc22ba410d33" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:47.472601 29735 raft_consensus.cc:2454] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 888d99522e4844118f8f1cf26f268c8c in term 5.
W20240501 20:01:47.476506 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 48444.1us
I20240501 20:01:47.444295 30944 raft_consensus.cc:482] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:47.447124 31453 raft_consensus.cc:482] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:47.480731 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e49dd5180f04313ab33942fb3ddc147" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:47.488668 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c597f15e844f30837012baed4b0501" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:47.494027 29809 raft_consensus.cc:2379] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 5: Already voted for candidate ee0d97247d394a8facfa774f15f3a67d in this term.
I20240501 20:01:47.495362 29624 leader_election.cc:302] T 1ca3948261374cef8b1b206a43c00173 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
W20240501 20:01:47.498390 29606 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet ece8d4ed84c34b748f7b6272b31c90ac, TS ee0d97247d394a8facfa774f15f3a67d -> TS 888d99522e4844118f8f1cf26f268c8c move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:47.498787 29606 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet ece8d4ed84c34b748f7b6272b31c90ac, TS ee0d97247d394a8facfa774f15f3a67d -> TS 888d99522e4844118f8f1cf26f268c8c move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:47.500514 29624 leader_election.cc:302] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:47.501452 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c597f15e844f30837012baed4b0501" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:47.407778 31471 raft_consensus.cc:504] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:47.495460 29700 leader_election.cc:302] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:47.503248 29624 leader_election.cc:302] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:47.482818 31312 raft_consensus.cc:2781] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:01:47.493471 31043 raft_consensus.cc:482] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240501 20:01:47.513732 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 91698.1us
I20240501 20:01:47.515264 29698 leader_election.cc:302] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:47.444051 31411 consensus_queue.cc:237] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:47.502339 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e07220db7444ae97f294d70eda6a93" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:47.532220 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668b4a4d782c4278932dd8413dc3ab3e" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:47.535429 29731 raft_consensus.cc:2454] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 4.
I20240501 20:01:47.534765 29698 leader_election.cc:302] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:47.508926 31427 raft_consensus.cc:2726] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240501 20:01:47.507165 31435 raft_consensus.cc:2726] T 1ca3948261374cef8b1b206a43c00173 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:01:47.544117 29772 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 45030.5us
I20240501 20:01:47.489861 30944 raft_consensus.cc:504] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
W20240501 20:01:47.549969 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 51656.4us
I20240501 20:01:47.510664 31449 raft_consensus.cc:3037] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:47.549382 29698 leader_election.cc:302] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:47.551774 29803 raft_consensus.cc:2396] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20240501 20:01:47.494783 31453 raft_consensus.cc:504] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:47.537662 31205 raft_consensus.cc:2726] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:47.554975 29698 leader_election.cc:302] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:47.537787 31460 raft_consensus.cc:482] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:47.535076 31467 raft_consensus.cc:482] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:47.538841 31312 raft_consensus.cc:686] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 5 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:47.557781 29773 leader_election.cc:302] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:47.558435 31487 consensus_queue.cc:1035] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 20:01:47.558508 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668b4a4d782c4278932dd8413dc3ab3e" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:47.559016 31486 raft_consensus.cc:2726] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:47.559531 29698 leader_election.cc:302] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:47.725098 29809 raft_consensus.cc:1261] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240501 20:01:47.561632 29801 raft_consensus.cc:1261] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240501 20:01:47.726671 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d2cac25f354a97b1bfb53d57ae7c08" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:47.627756 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "696121417ea9403990456cfc6fc4f416" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:47.728801 29731 raft_consensus.cc:2454] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 888d99522e4844118f8f1cf26f268c8c in term 4.
I20240501 20:01:47.730327 30384 leader_election.cc:302] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 888d99522e4844118f8f1cf26f268c8c, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:47.543347 31307 consensus_queue.cc:1035] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.018s
I20240501 20:01:47.620440 31413 leader_election.cc:288] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:47.552642 31432 leader_election.cc:288] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:47.553820 31474 log.cc:924] Time spent T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 0.161s user 0.000s sys 0.001s
I20240501 20:01:47.619725 31267 leader_election.cc:288] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:47.510905 31472 consensus_queue.cc:1035] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.049s
I20240501 20:01:47.582521 31467 raft_consensus.cc:504] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:47.585590 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 69649.9us
I20240501 20:01:47.727223 31428 raft_consensus.cc:2726] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 20:01:47.617926 31477 log.cc:924] Time spent T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 0.150s user 0.000s sys 0.000s
I20240501 20:01:47.668996 29727 raft_consensus.cc:1226] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 3. Current term is 4. Ops: []
I20240501 20:01:47.591653 29730 raft_consensus.cc:2379] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 5: Already voted for candidate ee0d97247d394a8facfa774f15f3a67d in this term.
I20240501 20:01:47.556434 31087 raft_consensus.cc:2726] T bfac893b4b4f457694a4eb184b6abb53 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240501 20:01:47.545130 29729 raft_consensus.cc:2454] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 888d99522e4844118f8f1cf26f268c8c in term 3.
W20240501 20:01:47.616780 31469 log.cc:924] Time spent T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 0.137s user 0.001s sys 0.000s
W20240501 20:01:47.626627 31468 log.cc:924] Time spent T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 0.382s user 0.000s sys 0.001s
I20240501 20:01:47.559338 31466 consensus_queue.cc:1035] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.180s
I20240501 20:01:47.561668 31460 raft_consensus.cc:504] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:47.619086 31483 log.cc:924] Time spent T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20240501 20:01:47.560014 31230 raft_consensus.cc:2726] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240501 20:01:47.627346 31448 leader_election.cc:288] T 1ca3948261374cef8b1b206a43c00173 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:47.599128 31197 raft_consensus.cc:504] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:47.532783 31043 raft_consensus.cc:504] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:47.723106 29807 raft_consensus.cc:2454] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 4.
W20240501 20:01:47.724210 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 166487us
I20240501 20:01:47.615962 31461 raft_consensus.cc:2781] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election won for term 3
W20240501 20:01:47.734839 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 175339us
I20240501 20:01:47.797268 29735 raft_consensus.cc:2454] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 5.
I20240501 20:01:47.674690 30944 leader_election.cc:288] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:47.676738 31436 raft_consensus.cc:482] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:47.685902 31471 leader_election.cc:288] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:47.750845 31449 raft_consensus.cc:2726] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20240501 20:01:47.730195 31442 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.552s user 0.011s sys 0.003s
I20240501 20:01:47.753088 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e49dd5180f04313ab33942fb3ddc147" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:47.737010 31445 raft_consensus.cc:482] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 888d99522e4844118f8f1cf26f268c8c)
I20240501 20:01:47.811110 31445 raft_consensus.cc:504] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:47.812280 31440 consensus_queue.cc:1035] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:47.813127 31043 leader_election.cc:288] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:47.813452 31445 leader_election.cc:288] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:47.741441 31486 raft_consensus.cc:2726] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240501 20:01:47.813555 31440 consensus_queue.cc:1035] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:47.764681 29805 raft_consensus.cc:1261] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240501 20:01:47.773806 29731 raft_consensus.cc:1261] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240501 20:01:47.815284 31506 raft_consensus.cc:482] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:47.779112 29733 raft_consensus.cc:1261] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240501 20:01:47.735662 31435 consensus_queue.cc:1035] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
I20240501 20:01:47.816613 31491 consensus_queue.cc:1035] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.058s
W20240501 20:01:47.818065 31258 consensus_meta.cc:287] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f: Time spent flushing consensus metadata: real 0.559s user 0.010s sys 0.000s
I20240501 20:01:47.818327 31467 leader_election.cc:288] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:47.677040 31312 consensus_queue.cc:237] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:47.818449 31258 raft_consensus.cc:504] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:47.819305 31495 log.cc:924] Time spent T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20240501 20:01:47.758543 31453 leader_election.cc:288] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:47.799046 29801 raft_consensus.cc:1261] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20240501 20:01:47.820389 29801 pending_rounds.cc:76] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d: Aborting all ops after (but not including) 2
I20240501 20:01:47.820662 29801 pending_rounds.cc:97] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20240501 20:01:47.820890 29801 raft_consensus.cc:2866] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 20:01:47.779585 31466 consensus_queue.cc:1225] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer fdfeac4190e045f290e003b2298d988d log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20240501 20:01:47.802101 29625 leader_election.cc:302] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
W20240501 20:01:47.823292 31450 consensus_meta.cc:287] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c: Time spent flushing consensus metadata: real 0.528s user 0.007s sys 0.000s
I20240501 20:01:47.792027 31444 raft_consensus.cc:2781] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240501 20:01:47.808427 29800 raft_consensus.cc:1226] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 5. Current term is 6. Ops: []
I20240501 20:01:47.787030 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d2cac25f354a97b1bfb53d57ae7c08" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
W20240501 20:01:47.810441 31442 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.632s user 0.011s sys 0.003s
I20240501 20:01:47.827899 29801 raft_consensus.cc:3032] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [term 4 LEADER]: Stepping down as leader of term 4
I20240501 20:01:47.835515 29801 raft_consensus.cc:729] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:47.823767 31153 raft_consensus.cc:2781] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240501 20:01:47.833405 29773 leader_election.cc:302] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 888d99522e4844118f8f1cf26f268c8c, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:47.844066 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07bdb8b09b8c4ef4b8e49daa9b8c01d6" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:47.847764 29726 raft_consensus.cc:1226] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [term 2 LEADER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 1. Current term is 2. Ops: []
I20240501 20:01:47.844993 29809 raft_consensus.cc:2454] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 4.
I20240501 20:01:47.830261 31444 raft_consensus.cc:482] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:47.823700 31450 raft_consensus.cc:504] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:47.815759 31506 raft_consensus.cc:504] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:47.864334 29773 leader_election.cc:302] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:47.835342 31411 raft_consensus.cc:482] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:47.855687 29700 leader_election.cc:302] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:47.847337 31153 raft_consensus.cc:482] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:47.866396 31428 consensus_queue.cc:1035] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:47.866106 31501 raft_consensus.cc:2726] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:47.867190 31413 raft_consensus.cc:482] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:47.819958 31439 raft_consensus.cc:504] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:47.869192 31210 raft_consensus.cc:2781] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240501 20:01:47.871220 29773 leader_election.cc:302] T a8787bc4d6b44649b9138d906730110a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:47.852838 31367 raft_consensus.cc:2726] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [term 2 LEADER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 20:01:47.872478 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 74579.6us
I20240501 20:01:47.846691 29801 consensus_queue.cc:260] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:47.830462 31432 consensus_queue.cc:1035] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.035s
I20240501 20:01:47.869853 30414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add9d7579351483fb268f89a44e95d8a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c" is_pre_election: true
I20240501 20:01:47.835808 31063 consensus_queue.cc:1035] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20240501 20:01:47.875455 30417 raft_consensus.cc:3037] T be383be783234855878cb8fd442c7b05 P 888d99522e4844118f8f1cf26f268c8c [term 3 LEARNER]: Advancing to term 4
I20240501 20:01:47.873358 31411 raft_consensus.cc:504] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } }
I20240501 20:01:47.880568 31509 consensus_queue.cc:1035] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.010s
I20240501 20:01:47.865250 31444 raft_consensus.cc:3037] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:47.809021 31436 raft_consensus.cc:504] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:47.885741 29801 raft_consensus.cc:3037] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:47.886371 29624 leader_election.cc:302] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:47.873526 31153 raft_consensus.cc:3037] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:47.889019 31333 consensus_queue.cc:1035] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:01:47.882854 31210 raft_consensus.cc:482] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:47.909943 31506 leader_election.cc:288] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:47.910281 31210 raft_consensus.cc:3037] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:47.910981 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e49dd5180f04313ab33942fb3ddc147" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:47.911679 29652 raft_consensus.cc:2454] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 4.
I20240501 20:01:47.893579 31493 raft_consensus.cc:482] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:47.867558 31498 raft_consensus.cc:482] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:47.874547 31472 raft_consensus.cc:3037] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:47.898447 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd5e83faaff4e2da4e4cc22ba410d33" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:47.901024 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07bdb8b09b8c4ef4b8e49daa9b8c01d6" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:47.874905 31258 leader_election.cc:288] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:47.894611 31435 consensus_queue.cc:1035] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.020s
I20240501 20:01:47.909461 30415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd5e83faaff4e2da4e4cc22ba410d33" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c" is_pre_election: true
I20240501 20:01:47.915083 29773 leader_election.cc:302] T 84fa28fb55f44b61ba1a88ed775a78ba P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:47.915673 29654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e07220db7444ae97f294d70eda6a93" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:47.906265 31501 raft_consensus.cc:2726] T a8787bc4d6b44649b9138d906730110a P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:47.927517 29654 raft_consensus.cc:2379] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d in current term 4: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:47.928205 29733 raft_consensus.cc:1261] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240501 20:01:47.928328 29654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48fd0a8e289401aa7417c42ff2c81b2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:47.929435 29654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add9d7579351483fb268f89a44e95d8a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:47.930349 29654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07bdb8b09b8c4ef4b8e49daa9b8c01d6" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:47.932262 31450 leader_election.cc:288] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 7 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:47.923354 31486 raft_consensus.cc:482] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:47.935104 31501 consensus_queue.cc:1035] T 6797d82dda2948d8a7445fcd33e44752 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:47.935220 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e38a2e91a3d48d09a22abce723af833" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:47.935950 29728 raft_consensus.cc:3037] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:47.936333 31501 consensus_queue.cc:1046] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:47.936910 31501 consensus_queue.cc:1225] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Peer fdfeac4190e045f290e003b2298d988d log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20240501 20:01:47.937043 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07bdb8b09b8c4ef4b8e49daa9b8c01d6" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:47.937461 31445 raft_consensus.cc:3032] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:01:47.937762 31445 raft_consensus.cc:729] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:47.937821 31197 leader_election.cc:288] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 6 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:47.923718 29659 raft_consensus.cc:2379] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d in current term 6: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:47.918649 29656 raft_consensus.cc:2454] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 5.
I20240501 20:01:47.908038 31205 raft_consensus.cc:482] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:47.884361 31413 raft_consensus.cc:504] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:47.938575 31445 consensus_queue.cc:260] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:47.946806 30415 raft_consensus.cc:2354] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d for earlier term 6. Current term is 7.
I20240501 20:01:47.947690 31513 raft_consensus.cc:482] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:47.950631 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd5e83faaff4e2da4e4cc22ba410d33" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:47.959869 29656 raft_consensus.cc:3037] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:01:47.911037 31453 raft_consensus.cc:482] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:47.954819 29773 leader_election.cc:302] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:47.957023 29729 raft_consensus.cc:1261] T c651044a305c4168805825a53d910899 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240501 20:01:47.948876 30418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e07220db7444ae97f294d70eda6a93" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c" is_pre_election: true
I20240501 20:01:47.932271 31307 raft_consensus.cc:2726] T 84fa28fb55f44b61ba1a88ed775a78ba P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:47.958926 29733 pending_rounds.cc:76] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d: Aborting all ops after (but not including) 2
I20240501 20:01:47.966579 29731 raft_consensus.cc:1261] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240501 20:01:47.974984 29733 pending_rounds.cc:97] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20240501 20:01:47.975970 29772 leader_election.cc:302] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, 888d99522e4844118f8f1cf26f268c8c
I20240501 20:01:47.976559 29733 raft_consensus.cc:2866] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 20:01:47.977954 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd5e83faaff4e2da4e4cc22ba410d33" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:47.978801 29729 raft_consensus.cc:3037] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Advancing to term 7
I20240501 20:01:47.955685 31440 raft_consensus.cc:482] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:47.951184 31461 raft_consensus.cc:686] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [term 3 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:48.008972 29728 raft_consensus.cc:2454] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 5.
W20240501 20:01:48.002197 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 47424.5us
W20240501 20:01:48.007005 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 53301.9us
I20240501 20:01:48.012600 29773 leader_election.cc:302] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:47.984421 31445 raft_consensus.cc:3037] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:47.952720 31205 raft_consensus.cc:504] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:48.034061 29731 raft_consensus.cc:1261] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20240501 20:01:47.925473 31493 raft_consensus.cc:504] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:47.944375 31486 raft_consensus.cc:504] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:48.036285 29656 raft_consensus.cc:2454] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 888d99522e4844118f8f1cf26f268c8c in term 7.
I20240501 20:01:48.028908 30418 raft_consensus.cc:2354] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d for earlier term 4. Current term is 5.
I20240501 20:01:48.034688 29729 raft_consensus.cc:2454] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 888d99522e4844118f8f1cf26f268c8c in term 7.
I20240501 20:01:47.942873 31498 raft_consensus.cc:504] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:48.051107 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e07220db7444ae97f294d70eda6a93" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:48.051740 29659 raft_consensus.cc:3037] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:48.023664 31436 leader_election.cc:288] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:48.014117 31063 consensus_queue.cc:1035] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20240501 20:01:47.969637 31500 consensus_queue.cc:1035] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.081s
I20240501 20:01:47.970541 31453 raft_consensus.cc:504] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:48.039510 31524 raft_consensus.cc:482] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:48.067301 30384 leader_election.cc:302] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 888d99522e4844118f8f1cf26f268c8c, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:48.067574 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69915dd5992641e3a24f7f45f1cd5c0c" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:48.052551 31197 raft_consensus.cc:2726] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240501 20:01:47.965610 31513 raft_consensus.cc:504] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:48.048883 31460 leader_election.cc:288] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:48.058091 31428 raft_consensus.cc:482] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:48.059346 31451 raft_consensus.cc:482] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:48.084497 29624 leader_election.cc:302] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:48.019587 31440 raft_consensus.cc:504] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:48.062420 31439 leader_election.cc:288] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 5 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
W20240501 20:01:48.095342 31516 log.cc:924] Time spent T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
I20240501 20:01:48.095769 31443 raft_consensus.cc:2781] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:01:48.096320 31443 raft_consensus.cc:686] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [term 5 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:48.097424 30417 raft_consensus.cc:1261] T be383be783234855878cb8fd442c7b05 P 888d99522e4844118f8f1cf26f268c8c [term 4 LEARNER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240501 20:01:48.097230 31443 consensus_queue.cc:237] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:48.058672 31471 consensus_queue.cc:1035] T c651044a305c4168805825a53d910899 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.029s
I20240501 20:01:48.079986 31411 leader_election.cc:288] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:48.086644 31450 raft_consensus.cc:2781] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [term 7 FOLLOWER]: Leader election won for term 7
I20240501 20:01:48.099200 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e38a2e91a3d48d09a22abce723af833" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:48.089617 31428 raft_consensus.cc:504] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:48.108793 31466 raft_consensus.cc:2781] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:01:48.100112 31413 leader_election.cc:288] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:48.128566 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c597f15e844f30837012baed4b0501" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:48.130162 29809 raft_consensus.cc:2454] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 5.
I20240501 20:01:48.111836 31506 raft_consensus.cc:482] T 3c5345f6eb62449ba69b4a9c2e349b31 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:48.131232 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e89c0572e4ee7be8e5315b459bdb3" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:48.134685 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07bdb8b09b8c4ef4b8e49daa9b8c01d6" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:48.135090 29809 raft_consensus.cc:2454] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 4.
I20240501 20:01:48.039403 31461 consensus_queue.cc:237] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:48.136408 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.196:37227, user_credentials={real_user=slave}} blocked reactor thread for 195767us
I20240501 20:01:48.136755 29625 leader_election.cc:302] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:48.117659 31492 raft_consensus.cc:2726] T add9d7579351483fb268f89a44e95d8a P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:01:48.133177 31462 log.cc:924] Time spent T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.897s user 0.004s sys 0.001s
I20240501 20:01:48.142043 31525 raft_consensus.cc:482] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:48.145610 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211e27a3b0084b89a6dd7550849b5a65" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:48.113165 31509 consensus_queue.cc:1035] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
I20240501 20:01:48.122561 31450 raft_consensus.cc:686] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [term 7 LEADER]: Becoming Leader. State: Replica: 888d99522e4844118f8f1cf26f268c8c, State: Running, Role: LEADER
I20240501 20:01:48.144456 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a35893b91247a99912eba787d787cf" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:48.079509 31524 raft_consensus.cc:504] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:48.153151 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48fd0a8e289401aa7417c42ff2c81b2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:48.153219 31438 leader_election.cc:288] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:48.178562 29801 raft_consensus.cc:1261] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240501 20:01:48.181706 31523 consensus_queue.cc:1035] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.111s
I20240501 20:01:48.183971 29734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 937) took 1663 ms. Trace:
I20240501 20:01:48.184423 31465 consensus_queue.cc:1035] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.069s
I20240501 20:01:48.090796 31451 raft_consensus.cc:504] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:48.185194 31517 consensus_queue.cc:1035] T 8edcd903224d48f889b46e1b5318e0ec P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.124s
I20240501 20:01:48.185564 31486 leader_election.cc:288] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:48.186189 31508 log.cc:924] Time spent T 458740ab5c444b6ab3d1991c69db5084 P ee0d97247d394a8facfa774f15f3a67d: Append to log took a long time: real 0.161s user 0.001s sys 0.000s
I20240501 20:01:48.186667 29654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c597f15e844f30837012baed4b0501" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:48.210245 29654 raft_consensus.cc:2454] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 4.
I20240501 20:01:48.186316 31153 raft_consensus.cc:504] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:48.207504 29734 rpcz_store.cc:276] 0501 20:01:46.520952 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:46.593469 (+ 72517us) service_pool.cc:225] Handling call
0501 20:01:46.610435 (+ 16966us) raft_consensus.cc:1457] Updating replica for 1 ops
0501 20:01:46.716030 (+105595us) raft_consensus.cc:1520] Early marking committed up to index 2
0501 20:01:46.951664 (+235634us) log.cc:842] Serialized 14 byte log entry
0501 20:01:47.403705 (+452041us) spinlock_profiling.cc:243] Waited 2.19 ms on lock 0x7b18003557e0. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe45d7f2661 00007fe45d7ff3e1 00007fe4548ca2ea 00007fe45dfb9fe5 00007fe45494e3d4 00007fe45494dcc5 00007fe457873620 00007fe4578857c6 00007fe457896327 00007fe4578962a8 00007fe457896228
0501 20:01:47.478589 (+ 74884us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0501 20:01:47.479434 (+ 845us) log.cc:842] Serialized 26 byte log entry
0501 20:01:47.479547 (+ 113us) raft_consensus.cc:1631] Marking committed up to 2
0501 20:01:47.479582 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:47.479640 (+ 58us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0501 20:01:47.957495 (+477855us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b44000232cc. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe454797b51 00007fe454865685 00007fe4548660c2 00007fe45750e441 00007fe457509d4a 00007fe4574fdb1e 00007fe4574fce66 00007fe4574fca88 00007fe457506481 00007fe457506439 00007fe4575063c9
0501 20:01:48.183729 (+226234us) raft_consensus.cc:1671] finished
0501 20:01:48.183765 (+ 36us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:48.183900 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":120703,"prepare.queue_time_us":258424,"prepare.run_cpu_time_us":78,"prepare.run_wall_time_us":2901,"raft.queue_time_us":395912,"raft.run_cpu_time_us":287,"raft.run_wall_time_us":59380,"spinlock_wait_cycles":31260928,"thread_start_us":389557,"threads_started":4,"wal-append.queue_time_us":162483}
I20240501 20:01:48.213220 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69915dd5992641e3a24f7f45f1cd5c0c" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:48.213523 31153 leader_election.cc:288] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:48.187738 31498 leader_election.cc:288] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:48.215253 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07bdb8b09b8c4ef4b8e49daa9b8c01d6" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:48.188077 29698 leader_election.cc:302] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:48.202829 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5525fc4b82a44b0991c3dd6098fde81" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:48.216800 29656 raft_consensus.cc:2379] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 5: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:48.136008 31466 raft_consensus.cc:686] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [term 5 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:48.143285 31436 raft_consensus.cc:2781] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240501 20:01:48.217633 31436 raft_consensus.cc:482] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:48.218008 31436 raft_consensus.cc:3037] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:48.218107 31466 consensus_queue.cc:237] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:48.223316 31486 raft_consensus.cc:2781] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240501 20:01:48.223752 31486 raft_consensus.cc:482] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240501 20:01:48.209111 31455 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.948s user 0.003s sys 0.007s
I20240501 20:01:48.224252 31486 raft_consensus.cc:3037] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Advancing to term 6
W20240501 20:01:48.224540 31455 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.964s user 0.003s sys 0.007s
I20240501 20:01:48.230727 31436 raft_consensus.cc:504] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:48.237049 31436 leader_election.cc:288] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:48.238200 29698 leader_election.cc:302] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:48.239326 31518 raft_consensus.cc:2781] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240501 20:01:48.239423 29698 leader_election.cc:398] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 pre-election: Vote denied by peer 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227) with higher term. Message: Invalid argument: T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d for earlier term 6. Current term is 7.
I20240501 20:01:48.239845 31518 raft_consensus.cc:482] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:48.240306 31518 raft_consensus.cc:3037] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:48.249562 29801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:48.252835 31521 log.cc:924] Time spent T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d: Append to log took a long time: real 0.162s user 0.000s sys 0.001s
I20240501 20:01:48.147599 31205 leader_election.cc:288] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:48.247553 29698 leader_election.cc:401] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20240501 20:01:48.263749 29658 raft_consensus.cc:1226] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [term 5 LEADER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 4. Current term is 5. Ops: []
I20240501 20:01:48.138060 31506 raft_consensus.cc:504] T 3c5345f6eb62449ba69b4a9c2e349b31 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:48.161198 31525 raft_consensus.cc:504] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:48.282421 31503 log.cc:924] Time spent T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d: Append to log took a long time: real 0.516s user 0.002s sys 0.000s
I20240501 20:01:48.283264 31525 leader_election.cc:288] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:48.286736 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.196:37227, user_credentials={real_user=slave}} blocked reactor thread for 47441.7us
I20240501 20:01:48.201815 30385 leader_election.cc:302] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 888d99522e4844118f8f1cf26f268c8c; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:48.296250 29698 leader_election.cc:398] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Vote denied by peer 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227) with higher term. Message: Invalid argument: T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d for earlier term 4. Current term is 5.
I20240501 20:01:48.297845 29698 leader_election.cc:401] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20240501 20:01:48.301720 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211e27a3b0084b89a6dd7550849b5a65" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:48.277516 31493 leader_election.cc:288] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:48.300846 29773 leader_election.cc:302] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:48.312521 31440 leader_election.cc:288] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:48.312567 29656 raft_consensus.cc:2454] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 3.
I20240501 20:01:48.295354 31450 consensus_queue.cc:237] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:48.314286 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1a08b455b94ae089396a15c837ddc2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:48.316677 31507 raft_consensus.cc:482] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:48.317174 31507 raft_consensus.cc:504] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:48.319526 31507 leader_election.cc:288] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:48.320211 31514 consensus_queue.cc:1035] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.073s
I20240501 20:01:48.321972 31292 raft_consensus.cc:2726] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240501 20:01:48.291136 31205 raft_consensus.cc:2726] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227) with higher term. Message: Invalid argument: T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d for earlier term 6. Current term is 7.
I20240501 20:01:48.334825 29698 leader_election.cc:302] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:48.336735 29773 leader_election.cc:302] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:48.337723 29658 raft_consensus.cc:1261] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240501 20:01:48.338348 29773 leader_election.cc:302] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:48.465193 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668b4a4d782c4278932dd8413dc3ab3e" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:48.465866 29804 raft_consensus.cc:3037] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:48.297506 31453 leader_election.cc:288] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:48.297291 29802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458740ab5c444b6ab3d1991c69db5084" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:48.471916 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 133645us
I20240501 20:01:48.269456 31486 raft_consensus.cc:504] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:48.477110 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be383be783234855878cb8fd442c7b05" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:48.477950 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3f29f0eeea4b119984634202e5d9cd" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:48.469027 31312 raft_consensus.cc:2726] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240501 20:01:48.359174 31525 raft_consensus.cc:3037] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Advancing to term 5
I20240501 20:01:48.469282 31411 raft_consensus.cc:2726] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240501 20:01:48.334755 31428 leader_election.cc:288] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:48.337317 31498 raft_consensus.cc:482] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:48.336385 31197 raft_consensus.cc:2726] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240501 20:01:48.481895 31500 raft_consensus.cc:482] T 3c5345f6eb62449ba69b4a9c2e349b31 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:48.482182 31492 raft_consensus.cc:482] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:48.530156 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6797d82dda2948d8a7445fcd33e44752" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
W20240501 20:01:48.476881 30385 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 315265us
W20240501 20:01:48.542115 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 44372.3us
I20240501 20:01:48.568255 29727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 1044) took 1406 ms. Trace:
I20240501 20:01:48.573083 29727 rpcz_store.cc:276] 0501 20:01:47.161623 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:47.161748 (+ 125us) service_pool.cc:225] Handling call
0501 20:01:47.162088 (+ 340us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:47.762040 (+599952us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:48.567954 (+805914us) spinlock_profiling.cc:243] Waited 507 ms on lock 0x7b54002305d4. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787c1a1 00007fe45787abee 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc 00007fe455e9e782 00007fe455ca2bcc 00007fe455ca2006
0501 20:01:48.568185 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1165340416}
I20240501 20:01:48.574173 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5525fc4b82a44b0991c3dd6098fde81" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:48.473852 31518 raft_consensus.cc:504] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:48.570423 31492 raft_consensus.cc:504] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:48.577697 31543 raft_consensus.cc:482] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240501 20:01:48.574784 31210 consensus_meta.cc:287] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d: Time spent flushing consensus metadata: real 0.662s user 0.011s sys 0.000s
I20240501 20:01:48.536334 31498 raft_consensus.cc:504] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:48.536059 31428 raft_consensus.cc:482] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:48.503952 31513 leader_election.cc:288] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:48.591902 29653 raft_consensus.cc:1226] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 5 LEADER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20240501 20:01:48.598639 29653 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59136 (request call id 881) took 1469 ms. Trace:
I20240501 20:01:48.579368 31212 raft_consensus.cc:482] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:48.599054 29653 rpcz_store.cc:276] 0501 20:01:47.129121 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:47.132872 (+ 3751us) service_pool.cc:225] Handling call
0501 20:01:48.591469 (+1458597us) spinlock_profiling.cc:243] Waited 28.5 ms on lock 0x7b5800001598. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe454797b51 00007fe4549193b1 00007fe454919262 00007fe45d80b889 00007fe45c7483c0 00007fe45c80e3bb 00007fe45c80e441 00007fe45c7d2060 00007fe45c7bd5a9 00007fe45c7bd052 00007fe455e9f694
0501 20:01:48.591657 (+ 188us) raft_consensus.cc:1457] Updating replica for 1 ops
0501 20:01:48.598306 (+ 6649us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:48.598538 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":65618816}
I20240501 20:01:48.599534 31212 raft_consensus.cc:504] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:48.588582 31428 raft_consensus.cc:504] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:48.571800 31453 raft_consensus.cc:2781] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:48.583251 31498 leader_election.cc:288] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:48.594138 31492 leader_election.cc:288] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:48.603169 29730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 1054) took 1154 ms. Trace:
W20240501 20:01:48.607028 29659 consensus_meta.cc:287] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f: Time spent flushing consensus metadata: real 0.555s user 0.008s sys 0.001s
I20240501 20:01:48.581161 31531 raft_consensus.cc:482] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:48.588100 31518 leader_election.cc:288] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:48.614424 29659 raft_consensus.cc:2454] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 888d99522e4844118f8f1cf26f268c8c in term 5.
I20240501 20:01:48.614315 29730 rpcz_store.cc:276] 0501 20:01:47.448994 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:47.467315 (+ 18321us) service_pool.cc:225] Handling call
0501 20:01:48.602953 (+1135638us) spinlock_profiling.cc:243] Waited 99.8 ms on lock 0x7b5400231ed4. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787f4fe 00007fe45c7bd9d7 00007fe455ea1c14 00007fe455ea1ba2 00007fe455ea1ad1 00007fe455ea1a4c 00007fe455ea0d02 00007fe455ca2bcc 00007fe455ca2006 00007fe455ca19af
0501 20:01:48.603098 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":229462912}
W20240501 20:01:48.627063 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 292164us
W20240501 20:01:48.595006 31445 consensus_meta.cc:287] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d: Time spent flushing consensus metadata: real 0.568s user 0.010s sys 0.000s
I20240501 20:01:48.599112 31524 leader_election.cc:288] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:48.580407 31210 raft_consensus.cc:504] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:48.605773 31428 leader_election.cc:288] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:48.584189 31526 raft_consensus.cc:482] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:48.580271 31543 raft_consensus.cc:504] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:48.631196 31531 raft_consensus.cc:504] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:48.632087 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e07220db7444ae97f294d70eda6a93" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:48.606027 31453 raft_consensus.cc:482] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:48.602320 31212 leader_election.cc:288] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:48.648883 29732 raft_consensus.cc:2379] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 6: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
W20240501 20:01:48.653188 31527 log.cc:924] Time spent T 1ca3948261374cef8b1b206a43c00173 P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 0.098s user 0.000s sys 0.000s
W20240501 20:01:48.653856 31532 log.cc:924] Time spent T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.051s user 0.001s sys 0.000s
W20240501 20:01:48.657464 31489 log.cc:924] Time spent T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d: Append to log took a long time: real 1.107s user 0.005s sys 0.011s
I20240501 20:01:48.649895 31453 raft_consensus.cc:3037] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:48.577191 31500 raft_consensus.cc:504] T 3c5345f6eb62449ba69b4a9c2e349b31 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:48.661026 29808 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41858 (request call id 1066) took 1244 ms. Trace:
I20240501 20:01:48.662274 31543 leader_election.cc:288] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:48.629703 31221 consensus_queue.cc:1046] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.022s
I20240501 20:01:48.644901 31524 raft_consensus.cc:482] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:48.673264 29727 raft_consensus.cc:2454] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 5.
I20240501 20:01:48.650287 31442 raft_consensus.cc:482] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:48.666460 29808 rpcz_store.cc:276] 0501 20:01:47.417383 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:47.418467 (+ 1084us) service_pool.cc:225] Handling call
0501 20:01:47.418818 (+ 351us) raft_consensus.cc:1457] Updating replica for 1 ops
0501 20:01:47.421670 (+ 2852us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:47.421696 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0501 20:01:47.425669 (+ 3973us) log.cc:842] Serialized 26 byte log entry
0501 20:01:47.549314 (+123645us) raft_consensus.cc:1631] Marking committed up to 2
0501 20:01:47.549343 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:47.549406 (+ 63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0501 20:01:48.656794 (+1107388us) spinlock_profiling.cc:243] Waited 1.11 ms on lock 0x7b44000232cc. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe454797b51 00007fe454865685 00007fe4548660c2 00007fe45750e441 00007fe457509d4a 00007fe4574fdb1e 00007fe4574fce66 00007fe4574fca88 00007fe457506481 00007fe457506439 00007fe4575063c9
0501 20:01:48.660627 (+ 3833us) raft_consensus.cc:1671] finished
0501 20:01:48.660670 (+ 43us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:48.660933 (+ 263us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2452,"raft.queue_time_us":2939,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":3658,"spinlock_wait_cycles":2560256,"thread_start_us":79574,"threads_started":1,"wal-append.queue_time_us":111004}
I20240501 20:01:48.678670 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:48.679374 29731 raft_consensus.cc:2454] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 5.
I20240501 20:01:48.680192 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07bdb8b09b8c4ef4b8e49daa9b8c01d6" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:48.681258 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211e27a3b0084b89a6dd7550849b5a65" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:48.683290 31212 raft_consensus.cc:482] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:48.687374 31212 raft_consensus.cc:504] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:48.689554 31221 consensus_queue.cc:1225] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Peer 4c8da1d580614449a636b855bf11b95f log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20240501 20:01:48.689987 31212 leader_election.cc:288] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:48.690295 31426 leader_election.cc:288] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:48.690341 31221 raft_consensus.cc:3032] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [term 4 LEADER]: Stepping down as leader of term 4
I20240501 20:01:48.691071 31221 raft_consensus.cc:729] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:48.685277 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07bdb8b09b8c4ef4b8e49daa9b8c01d6" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:48.692193 29808 raft_consensus.cc:3037] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Advancing to term 6
I20240501 20:01:48.692037 31221 consensus_queue.cc:260] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:48.692660 31531 leader_election.cc:288] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:48.693110 31221 raft_consensus.cc:3037] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:48.694730 31210 leader_election.cc:288] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:48.695827 31486 leader_election.cc:288] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:48.690457 31492 raft_consensus.cc:482] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:48.697579 31492 raft_consensus.cc:504] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:48.704429 31492 leader_election.cc:288] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:48.647202 31526 raft_consensus.cc:504] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:48.712800 31526 leader_election.cc:288] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:48.714344 31442 raft_consensus.cc:504] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:48.716965 31442 leader_election.cc:288] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:48.717943 31536 consensus_queue.cc:1035] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.086s
W20240501 20:01:48.724859 31481 log.cc:924] Time spent T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 1.214s user 0.003s sys 0.000s
I20240501 20:01:48.677232 31524 raft_consensus.cc:504] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:48.729153 31524 leader_election.cc:288] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:48.730616 31444 consensus_meta.cc:287] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c: Time spent flushing consensus metadata: real 0.818s user 0.007s sys 0.003s
I20240501 20:01:48.731194 31444 raft_consensus.cc:504] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:48.734829 31444 leader_election.cc:288] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 5 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:48.743009 31472 consensus_meta.cc:287] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d: Time spent flushing consensus metadata: real 0.801s user 0.009s sys 0.002s
I20240501 20:01:48.752215 31472 raft_consensus.cc:2726] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:01:48.741953 31453 raft_consensus.cc:504] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:48.749224 31500 leader_election.cc:288] T 3c5345f6eb62449ba69b4a9c2e349b31 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:48.746888 31525 raft_consensus.cc:2726] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227) with higher term. Message: Invalid argument: T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d for earlier term 4. Current term is 5.
I20240501 20:01:48.757797 31453 leader_election.cc:288] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:48.808738 31511 raft_consensus.cc:2726] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240501 20:01:48.816426 31451 leader_election.cc:288] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:48.874541 29804 raft_consensus.cc:2454] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 5.
I20240501 20:01:48.875377 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05df43bdfa479b902b1da297fb5974" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:48.875831 29804 raft_consensus.cc:2454] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 888d99522e4844118f8f1cf26f268c8c in term 5.
I20240501 20:01:48.876506 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9472c53f79ec423f82c396bd980674e2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:48.876602 30385 leader_election.cc:302] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 888d99522e4844118f8f1cf26f268c8c, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:48.877610 31444 raft_consensus.cc:2781] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240501 20:01:48.878260 31444 raft_consensus.cc:482] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [term 5 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:48.878854 31444 raft_consensus.cc:3037] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:48.876925 29804 raft_consensus.cc:2454] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 4.
I20240501 20:01:48.887091 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "696121417ea9403990456cfc6fc4f416" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:48.887897 29804 raft_consensus.cc:3037] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:48.886126 31444 raft_consensus.cc:504] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
W20240501 20:01:48.892396 31541 log.cc:924] Time spent T 38cbeb0dc0c14e3d9182101eea885625 P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.286s user 0.000s sys 0.004s
I20240501 20:01:48.897125 29804 raft_consensus.cc:2454] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 888d99522e4844118f8f1cf26f268c8c in term 5.
I20240501 20:01:48.898198 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05df43bdfa479b902b1da297fb5974" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:48.898303 30385 leader_election.cc:302] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 888d99522e4844118f8f1cf26f268c8c, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:48.899223 29804 raft_consensus.cc:3037] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:48.898686 31444 leader_election.cc:288] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 6 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:48.899855 31455 raft_consensus.cc:2781] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:01:48.900373 31455 raft_consensus.cc:686] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [term 5 LEADER]: Becoming Leader. State: Replica: 888d99522e4844118f8f1cf26f268c8c, State: Running, Role: LEADER
I20240501 20:01:48.901132 31455 consensus_queue.cc:237] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:48.906322 29804 raft_consensus.cc:2454] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 888d99522e4844118f8f1cf26f268c8c in term 6.
I20240501 20:01:48.907505 30385 leader_election.cc:302] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 888d99522e4844118f8f1cf26f268c8c, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:48.907755 29804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41858 (request call id 1090) took 1002 ms. Trace:
I20240501 20:01:48.908433 31444 raft_consensus.cc:2781] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [term 6 FOLLOWER]: Leader election won for term 6
I20240501 20:01:48.908144 29804 rpcz_store.cc:276] 0501 20:01:47.905341 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:48.907039 (+1001698us) service_pool.cc:225] Handling call
0501 20:01:48.907315 (+ 276us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:48.907463 (+ 148us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:48.907486 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:48.907524 (+ 38us) raft_consensus.cc:1631] Marking committed up to 2
0501 20:01:48.907561 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:48.907606 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:48.907711 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 20:01:48.910749 29804 raft_consensus.cc:1261] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240501 20:01:48.915022 31444 raft_consensus.cc:686] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [term 6 LEADER]: Becoming Leader. State: Replica: 888d99522e4844118f8f1cf26f268c8c, State: Running, Role: LEADER
I20240501 20:01:48.934479 29731 raft_consensus.cc:2379] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 4: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:48.937718 29804 raft_consensus.cc:1261] T 9046e38fc57f4364aa3d539e1fc01928 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Refusing update from remote peer 888d99522e4844118f8f1cf26f268c8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240501 20:01:48.966162 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9472c53f79ec423f82c396bd980674e2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:48.966966 29729 raft_consensus.cc:2379] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 4: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:48.967806 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05df43bdfa479b902b1da297fb5974" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:48.969043 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f261d2a05bf4c838532049601694b4a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:48.970127 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458740ab5c444b6ab3d1991c69db5084" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:48.971521 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a2bba62f434d1c8489b55c8fec2187" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:48.972105 29729 raft_consensus.cc:2454] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 3.
I20240501 20:01:48.973240 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:48.973994 29729 raft_consensus.cc:2454] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 6.
I20240501 20:01:48.974690 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0374bb790ed345ed90452595f71383a2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:48.975253 29729 raft_consensus.cc:2454] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 5.
I20240501 20:01:48.975420 29773 leader_election.cc:302] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:48.976366 29773 leader_election.cc:302] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:48.976649 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5345f6eb62449ba69b4a9c2e349b31" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:48.976972 31523 raft_consensus.cc:2781] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:48.978230 31523 raft_consensus.cc:482] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:48.978768 31549 raft_consensus.cc:2781] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240501 20:01:48.979962 31523 raft_consensus.cc:3037] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:48.980086 31549 raft_consensus.cc:482] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:48.980528 31549 raft_consensus.cc:3037] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:01:48.981916 29808 raft_consensus.cc:2454] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 6.
I20240501 20:01:48.979462 29773 leader_election.cc:302] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:48.987453 31549 raft_consensus.cc:504] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:48.978732 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05df43bdfa479b902b1da297fb5974" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:48.992298 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:48.992830 29729 raft_consensus.cc:3037] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:01:48.987560 31523 raft_consensus.cc:504] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:48.996013 31523 leader_election.cc:288] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:48.993104 31549 leader_election.cc:288] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 7 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:48.996655 31523 raft_consensus.cc:2781] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240501 20:01:48.997048 31523 raft_consensus.cc:482] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:48.998476 31523 raft_consensus.cc:3037] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:49.008262 31549 raft_consensus.cc:482] T 456035a1cb3f4f7da52d0fc1dd1c656c P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:49.009099 29729 raft_consensus.cc:2454] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 7.
I20240501 20:01:49.009649 31488 raft_consensus.cc:482] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:49.010316 31549 raft_consensus.cc:504] T 456035a1cb3f4f7da52d0fc1dd1c656c P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:49.010560 31488 raft_consensus.cc:504] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:49.012624 31488 leader_election.cc:288] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:49.013715 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05df43bdfa479b902b1da297fb5974" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:49.014976 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668b4a4d782c4278932dd8413dc3ab3e" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:49.015560 29729 raft_consensus.cc:3037] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:49.015256 31523 raft_consensus.cc:504] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:49.017526 31549 leader_election.cc:288] T 456035a1cb3f4f7da52d0fc1dd1c656c P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:49.018136 31523 leader_election.cc:288] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 6 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:49.019268 31488 raft_consensus.cc:482] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:49.019771 31488 raft_consensus.cc:504] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:49.021610 31488 leader_election.cc:288] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:49.022998 29773 leader_election.cc:302] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:49.023607 31492 raft_consensus.cc:2781] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 7 FOLLOWER]: Leader election won for term 7
I20240501 20:01:49.023981 31492 raft_consensus.cc:686] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 7 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:49.024649 31492 consensus_queue.cc:237] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:49.032651 29729 raft_consensus.cc:2454] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 5.
I20240501 20:01:49.033557 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "456035a1cb3f4f7da52d0fc1dd1c656c" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:49.034901 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a2bba62f434d1c8489b55c8fec2187" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:49.035387 29729 raft_consensus.cc:3037] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:48.924857 31444 consensus_queue.cc:237] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:49.042618 29729 raft_consensus.cc:2454] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 4.
I20240501 20:01:49.043524 29773 leader_election.cc:302] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:49.043731 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0374bb790ed345ed90452595f71383a2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:49.044309 29729 raft_consensus.cc:3037] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:49.044413 31492 raft_consensus.cc:2781] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:01:49.044946 31492 raft_consensus.cc:686] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [term 4 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:49.045759 31492 consensus_queue.cc:237] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:49.052228 29729 raft_consensus.cc:2454] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 6.
I20240501 20:01:49.053295 29773 leader_election.cc:302] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:49.054204 31549 raft_consensus.cc:2781] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Leader election won for term 6
I20240501 20:01:49.054665 31549 raft_consensus.cc:686] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 6 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:49.055431 31549 consensus_queue.cc:237] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:49.056895 29803 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41858 (request call id 1046) took 2252 ms (client timeout 2164 ms). Trace:
W20240501 20:01:49.057314 29803 rpcz_store.cc:269] 0501 20:01:46.804034 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:46.809322 (+ 5288us) service_pool.cc:225] Handling call
0501 20:01:49.056666 (+2247344us) spinlock_profiling.cc:243] Waited 721 ms on lock 0x7b5400211454. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787f4fe 00007fe45c7bd9d7 00007fe455ea1c14 00007fe455ea1ba2 00007fe455ea1ad1 00007fe455ea1a4c 00007fe455ea0d02 00007fe455ca2bcc 00007fe455ca2006 00007fe455ca19af
0501 20:01:49.056809 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1657697152}
I20240501 20:01:49.062021 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "696121417ea9403990456cfc6fc4f416" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:49.062819 29806 raft_consensus.cc:2362] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 888d99522e4844118f8f1cf26f268c8c in term 5. Re-sending same reply.
I20240501 20:01:49.067922 29726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 1045) took 1896 ms. Trace:
I20240501 20:01:49.071282 29726 rpcz_store.cc:276] 0501 20:01:47.171088 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:47.171382 (+ 294us) service_pool.cc:225] Handling call
0501 20:01:47.177002 (+ 5620us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:47.850905 (+673903us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:49.067664 (+1216759us) spinlock_profiling.cc:243] Waited 670 ms on lock 0x7b54003700d4. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787c1a1 00007fe45787abee 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc 00007fe455e9e782 00007fe455ca2bcc 00007fe455ca2006
0501 20:01:49.067847 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1542112128}
I20240501 20:01:49.074714 29651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59142 (request call id 1007) took 1170 ms. Trace:
I20240501 20:01:49.074823 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "696121417ea9403990456cfc6fc4f416" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:49.075452 29726 raft_consensus.cc:3037] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:49.075163 29651 rpcz_store.cc:276] 0501 20:01:47.904374 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:47.907098 (+ 2724us) service_pool.cc:225] Handling call
0501 20:01:47.907480 (+ 382us) raft_consensus.cc:1457] Updating replica for 1 ops
0501 20:01:47.997950 (+ 90470us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:48.002469 (+ 4519us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0501 20:01:48.046168 (+ 43699us) log.cc:842] Serialized 26 byte log entry
0501 20:01:48.125602 (+ 79434us) raft_consensus.cc:1631] Marking committed up to 2
0501 20:01:48.132967 (+ 7365us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:49.074310 (+941343us) spinlock_profiling.cc:243] Waited 9.23 ms on lock 0x7b5400133b54. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787c1a1 00007fe45787abee 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc 00007fe455e9e782 00007fe455ca2bcc 00007fe455ca2006
0501 20:01:49.074359 (+ 49us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0501 20:01:49.074415 (+ 56us) raft_consensus.cc:1671] finished
0501 20:01:49.074430 (+ 15us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:49.074662 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":40746,"raft.queue_time_us":47727,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":17227,"spinlock_wait_cycles":21233408,"thread_start_us":29519,"threads_started":1,"wal-append.queue_time_us":116344}
W20240501 20:01:49.079478 29735 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 1055) took 1605 ms (client timeout 2103 ms). Trace:
W20240501 20:01:49.079849 29735 rpcz_store.cc:269] 0501 20:01:47.474224 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:47.481975 (+ 7751us) service_pool.cc:225] Handling call
0501 20:01:49.079332 (+1597357us) spinlock_profiling.cc:243] Waited 295 ms on lock 0x7b5400230ad4. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787f4fe 00007fe45c7bd9d7 00007fe455ea1c14 00007fe455ea1ba2 00007fe455ea1ad1 00007fe455ea1a4c 00007fe455ea0d02 00007fe455ca2bcc 00007fe455ca2006 00007fe455ca19af
0501 20:01:49.079428 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":679049600}
I20240501 20:01:49.083004 29726 raft_consensus.cc:2454] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 888d99522e4844118f8f1cf26f268c8c in term 5.
I20240501 20:01:49.084754 31506 leader_election.cc:288] T 3c5345f6eb62449ba69b4a9c2e349b31 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:49.085441 29660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59136 (request call id 874) took 2452 ms. Trace:
I20240501 20:01:49.085783 29660 rpcz_store.cc:276] 0501 20:01:46.632933 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:46.633071 (+ 138us) service_pool.cc:225] Handling call
0501 20:01:46.662353 (+ 29282us) raft_consensus.cc:1457] Updating replica for 1 ops
0501 20:01:46.892273 (+229920us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:46.892315 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0501 20:01:46.897070 (+ 4755us) log.cc:842] Serialized 26 byte log entry
0501 20:01:47.113892 (+216822us) raft_consensus.cc:1631] Marking committed up to 2
0501 20:01:47.113949 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:47.114236 (+ 287us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0501 20:01:47.401123 (+286887us) spinlock_profiling.cc:243] Waited 2.8 ms on lock 0x7b44000232cc. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe454797b51 00007fe454865685 00007fe4548660c2 00007fe45750e441 00007fe457509d4a 00007fe4574fdb1e 00007fe4574fce66 00007fe4574fca88 00007fe457506481 00007fe457506439 00007fe4575063c9
0501 20:01:49.078364 (+1677241us) spinlock_profiling.cc:243] Waited 4.03 ms on lock 0x7b240017ce28. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe454797b51 00007fe4547ac024 00007fe4547aa13b 00007fe454838149 00007fe454837dd5 00007fe454837330 00007fe45752a3a8 00007fe45752ab08 00007fe457502618 00007fe4574fda1a 00007fe4574fce66
0501 20:01:49.085258 (+ 6894us) raft_consensus.cc:1671] finished
0501 20:01:49.085278 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:49.085394 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":82030,"raft.queue_time_us":89929,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":14588,"spinlock_wait_cycles":15711488,"thread_start_us":258522,"threads_started":2,"wal-append.queue_time_us":284029}
I20240501 20:01:49.098858 31507 consensus_queue.cc:1035] T 9046e38fc57f4364aa3d539e1fc01928 P 888d99522e4844118f8f1cf26f268c8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240501 20:01:49.072908 29809 raft_consensus.cc:2454] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 3.
I20240501 20:01:49.115002 29773 leader_election.cc:302] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:49.116569 31221 raft_consensus.cc:2781] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240501 20:01:49.116989 31221 raft_consensus.cc:482] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:49.117370 31221 raft_consensus.cc:3037] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:49.129513 31221 raft_consensus.cc:504] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:49.135521 29805 raft_consensus.cc:1226] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 6 LEADER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 5. Current term is 6. Ops: []
I20240501 20:01:49.136694 31498 raft_consensus.cc:482] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:49.137212 31498 raft_consensus.cc:504] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:49.138231 29805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41858 (request call id 1086) took 1312 ms. Trace:
I20240501 20:01:49.138554 29805 rpcz_store.cc:276] 0501 20:01:47.826168 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:47.827188 (+ 1020us) service_pool.cc:225] Handling call
0501 20:01:49.135373 (+1308185us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:49.135986 (+ 613us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:49.138067 (+ 2081us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b5400211450. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787ac59 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc 00007fe455e9e782 00007fe455ca2bcc 00007fe455ca2006 00007fe455ca19af
0501 20:01:49.138181 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3007965568}
I20240501 20:01:49.139053 31498 leader_election.cc:288] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:49.132421 31221 leader_election.cc:288] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:49.157186 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0374bb790ed345ed90452595f71383a2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:49.161096 29651 raft_consensus.cc:3037] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:49.164999 29800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41858 (request call id 1068) took 1739 ms. Trace:
I20240501 20:01:49.165403 29800 rpcz_store.cc:276] 0501 20:01:47.425485 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:47.425624 (+ 139us) service_pool.cc:225] Handling call
0501 20:01:47.433435 (+ 7811us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:47.831405 (+397970us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:49.164789 (+1333384us) spinlock_profiling.cc:243] Waited 373 ms on lock 0x7b540031b854. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787c1a1 00007fe45787abee 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc 00007fe455e9e782 00007fe455ca2bcc 00007fe455ca2006
0501 20:01:49.164948 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":857859968}
I20240501 20:01:49.172405 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f261d2a05bf4c838532049601694b4a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:49.173029 29652 raft_consensus.cc:2454] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 4.
I20240501 20:01:49.173148 29728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 940) took 1033 ms. Trace:
I20240501 20:01:49.174243 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:49.173462 29728 rpcz_store.cc:276] 0501 20:01:48.139411 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:48.259689 (+120278us) service_pool.cc:225] Handling call
0501 20:01:48.274334 (+ 14645us) raft_consensus.cc:1457] Updating replica for 1 ops
0501 20:01:48.288491 (+ 14157us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:48.288518 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0501 20:01:48.288941 (+ 423us) log.cc:842] Serialized 26 byte log entry
0501 20:01:48.535898 (+246957us) raft_consensus.cc:1631] Marking committed up to 3
0501 20:01:48.572708 (+ 36810us) log.cc:842] Serialized 14 byte log entry
0501 20:01:48.642974 (+ 70266us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:48.643130 (+ 156us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0501 20:01:49.172966 (+529836us) raft_consensus.cc:1671] finished
0501 20:01:49.172985 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:49.173095 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":41405,"prepare.queue_time_us":36829,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"raft.queue_time_us":28170,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":4303,"thread_start_us":292064,"threads_started":2,"wal-append.queue_time_us":350237}
W20240501 20:01:49.172461 29775 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.695s user 0.018s sys 0.040s
W20240501 20:01:49.176615 29775 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.699s user 0.019s sys 0.040s
I20240501 20:01:49.179219 29801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a35893b91247a99912eba787d787cf" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:49.180955 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5525fc4b82a44b0991c3dd6098fde81" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
W20240501 20:01:49.182292 31540 log.cc:924] Time spent T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.709s user 0.004s sys 0.000s
I20240501 20:01:49.185240 29734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 939) took 1046 ms. Trace:
I20240501 20:01:49.185539 29734 rpcz_store.cc:276] 0501 20:01:48.139100 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:48.222775 (+ 83675us) service_pool.cc:225] Handling call
0501 20:01:48.230676 (+ 7901us) raft_consensus.cc:1457] Updating replica for 1 ops
0501 20:01:48.230889 (+ 213us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:48.230918 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0501 20:01:48.231394 (+ 476us) log.cc:842] Serialized 26 byte log entry
0501 20:01:48.256527 (+ 25133us) raft_consensus.cc:1631] Marking committed up to 4
0501 20:01:48.257159 (+ 632us) log.cc:842] Serialized 14 byte log entry
0501 20:01:48.257420 (+ 261us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:48.257515 (+ 95us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0501 20:01:49.181911 (+924396us) spinlock_profiling.cc:243] Waited 25.4 ms on lock 0x7b44000232cc. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe454797b51 00007fe454865685 00007fe4548660c2 00007fe45750e441 00007fe457509d4a 00007fe4574fdb1e 00007fe4574fce66 00007fe4574fca88 00007fe457506481 00007fe457506439 00007fe4575063c9
0501 20:01:49.184950 (+ 3039us) raft_consensus.cc:1671] finished
0501 20:01:49.185062 (+ 112us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:49.185192 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20600,"prepare.queue_time_us":1842,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":2696,"raft.queue_time_us":12778,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":126,"spinlock_wait_cycles":58451200,"thread_start_us":24633,"threads_started":1,"wal-append.queue_time_us":58945}
I20240501 20:01:49.187283 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5345f6eb62449ba69b4a9c2e349b31" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:49.187768 29728 raft_consensus.cc:2454] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 5.
I20240501 20:01:49.188666 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a2bba62f434d1c8489b55c8fec2187" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:49.191401 29660 raft_consensus.cc:1261] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240501 20:01:49.197181 29735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 941) took 1057 ms. Trace:
I20240501 20:01:49.192704 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af08384057a4d85977481472f460168" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:49.195874 29726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36260 (request call id 943) took 1055 ms. Trace:
I20240501 20:01:49.198814 29728 raft_consensus.cc:1226] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20240501 20:01:49.199633 29773 leader_election.cc:302] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:49.197659 29735 rpcz_store.cc:276] 0501 20:01:48.139703 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:49.080579 (+940876us) service_pool.cc:225] Handling call
0501 20:01:49.080915 (+ 336us) raft_consensus.cc:1457] Updating replica for 1 ops
0501 20:01:49.081243 (+ 328us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:49.081268 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0501 20:01:49.081659 (+ 391us) log.cc:842] Serialized 26 byte log entry
0501 20:01:49.127051 (+ 45392us) raft_consensus.cc:1631] Marking committed up to 3
0501 20:01:49.196437 (+ 69386us) log.cc:842] Serialized 14 byte log entry
0501 20:01:49.196796 (+ 359us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:49.196883 (+ 87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0501 20:01:49.196935 (+ 52us) raft_consensus.cc:1671] finished
0501 20:01:49.196952 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:49.197118 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":58920,"prepare.run_cpu_time_us":27,"prepare.run_wall_time_us":27,"raft.queue_time_us":1418,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":91,"thread_start_us":107742,"threads_started":2,"wal-append.queue_time_us":38331}
I20240501 20:01:49.203490 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649b71f9f267437ca51c758dd9ca6987" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:49.204700 29734 raft_consensus.cc:2454] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 5.
I20240501 20:01:49.198904 29726 rpcz_store.cc:276] 0501 20:01:48.140369 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:49.083657 (+943288us) service_pool.cc:225] Handling call
0501 20:01:49.083954 (+ 297us) raft_consensus.cc:1457] Updating replica for 1 ops
0501 20:01:49.084174 (+ 220us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:49.084199 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0501 20:01:49.084554 (+ 355us) log.cc:842] Serialized 26 byte log entry
0501 20:01:49.099277 (+ 14723us) spinlock_profiling.cc:243] Waited 5.33 ms on lock 0x7b54002abd20. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe457800d30 00007fe45780b268 00007fe45787bccb 00007fe45787abee 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc 00007fe455e9e782 00007fe455ca2bcc
0501 20:01:49.099333 (+ 56us) raft_consensus.cc:1631] Marking committed up to 3
0501 20:01:49.193587 (+ 94254us) log.cc:842] Serialized 14 byte log entry
0501 20:01:49.194165 (+ 578us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:49.194268 (+ 103us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0501 20:01:49.194324 (+ 56us) raft_consensus.cc:1671] finished
0501 20:01:49.194454 (+ 130us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:49.195816 (+ 1362us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":92765,"prepare.run_cpu_time_us":23,"prepare.run_wall_time_us":23,"raft.queue_time_us":129,"raft.run_cpu_time_us":71,"raft.run_wall_time_us":72,"spinlock_wait_cycles":12253952,"thread_start_us":102075,"threads_started":2,"wal-append.queue_time_us":6672}
I20240501 20:01:49.209638 29726 raft_consensus.cc:1226] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 5. Current term is 6. Ops: []
I20240501 20:01:49.210019 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3f29f0eeea4b119984634202e5d9cd" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:49.210664 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211e27a3b0084b89a6dd7550849b5a65" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:49.213575 29734 raft_consensus.cc:1226] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 4. Current term is 5. Ops: []
I20240501 20:01:49.221757 29727 raft_consensus.cc:1261] T 8bd5e83faaff4e2da4e4cc22ba410d33 P fdfeac4190e045f290e003b2298d988d [term 7 FOLLOWER]: Refusing update from remote peer 888d99522e4844118f8f1cf26f268c8c: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20240501 20:01:49.223809 31507 consensus_queue.cc:1035] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240501 20:01:49.227169 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be383be783234855878cb8fd442c7b05" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:49.217068 31472 raft_consensus.cc:2781] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:01:49.234033 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8edcd903224d48f889b46e1b5318e0ec" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:49.233372 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c597f15e844f30837012baed4b0501" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:49.235146 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1a08b455b94ae089396a15c837ddc2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:49.235634 29803 raft_consensus.cc:3037] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:49.237419 29734 raft_consensus.cc:1226] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 7 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 6. Current term is 7. Ops: [6.3-6.3]
I20240501 20:01:49.239902 29734 raft_consensus.cc:1226] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 3. Current term is 4. Ops: []
I20240501 20:01:49.221087 31498 consensus_queue.cc:1046] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
I20240501 20:01:49.255472 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6797d82dda2948d8a7445fcd33e44752" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:49.261607 29606 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 03f5d1686214485d91a0baf4b870740e, TS 4c8da1d580614449a636b855bf11b95f -> TS 888d99522e4844118f8f1cf26f268c8c move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:49.262066 29606 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 03f5d1686214485d91a0baf4b870740e, TS 4c8da1d580614449a636b855bf11b95f -> TS 888d99522e4844118f8f1cf26f268c8c move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:49.259061 29802 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.658s user 0.029s sys 0.028s
I20240501 20:01:49.266957 29803 raft_consensus.cc:2454] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 6.
I20240501 20:01:49.252710 31472 raft_consensus.cc:686] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [term 5 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:49.254734 29651 raft_consensus.cc:1261] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240501 20:01:49.269028 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8787bc4d6b44649b9138d906730110a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:49.270248 29806 raft_consensus.cc:2454] T a8787bc4d6b44649b9138d906730110a P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 3.
W20240501 20:01:49.263612 29802 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.663s user 0.030s sys 0.028s
I20240501 20:01:49.277818 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece8d4ed84c34b748f7b6272b31c90ac" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:49.284118 31467 raft_consensus.cc:482] T 48630ae6eb394ac1b54955dd01cf0ed2 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:49.294003 29802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41858 (request call id 1089) took 1394 ms. Trace:
I20240501 20:01:49.295544 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e49dd5180f04313ab33942fb3ddc147" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:49.295965 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07bdb8b09b8c4ef4b8e49daa9b8c01d6" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:49.296921 29805 raft_consensus.cc:2379] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d in current term 6: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:49.297415 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0374bb790ed345ed90452595f71383a2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:49.294416 29802 rpcz_store.cc:276] 0501 20:01:47.899272 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:48.579212 (+679940us) service_pool.cc:225] Handling call
0501 20:01:48.599829 (+ 20617us) raft_consensus.cc:1457] Updating replica for 1 ops
0501 20:01:48.600212 (+ 383us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:48.600245 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0501 20:01:48.600695 (+ 450us) log.cc:842] Serialized 26 byte log entry
0501 20:01:49.284503 (+683808us) raft_consensus.cc:1631] Marking committed up to 2
0501 20:01:49.285117 (+ 614us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:49.287358 (+ 2241us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0501 20:01:49.287897 (+ 539us) raft_consensus.cc:1671] finished
0501 20:01:49.288608 (+ 711us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:49.293240 (+ 4632us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1065,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":90,"thread_start_us":676483,"threads_started":1,"wal-append.queue_time_us":150063}
I20240501 20:01:49.299369 29733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 1068) took 1365 ms. Trace:
I20240501 20:01:49.299697 29733 rpcz_store.cc:276] 0501 20:01:47.934118 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:47.939402 (+ 5284us) service_pool.cc:225] Handling call
0501 20:01:47.958675 (+ 19273us) raft_consensus.cc:1457] Updating replica for 2 ops
0501 20:01:48.016635 (+ 57960us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:48.016678 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0501 20:01:49.163083 (+1146405us) spinlock_profiling.cc:243] Waited 1.01 ms on lock 0x7b2c00000a78. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe45625df11 00007fe45626d542 00007fe4578b6fc1 00007fe45787b7da 00007fe45787abee 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc 00007fe455e9e782
0501 20:01:49.163534 (+ 451us) log.cc:842] Serialized 52 byte log entry
0501 20:01:49.233362 (+ 69828us) raft_consensus.cc:1631] Marking committed up to 3
0501 20:01:49.255984 (+ 22622us) log.cc:842] Serialized 14 byte log entry
0501 20:01:49.257049 (+ 1065us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:49.257119 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0501 20:01:49.292330 (+ 35211us) raft_consensus.cc:1671] finished
0501 20:01:49.299201 (+ 6871us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:49.299325 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20189,"prepare.queue_time_us":9050,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"raft.queue_time_us":82756,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":4348,"spinlock_wait_cycles":2316672,"thread_start_us":69524,"threads_started":1,"wal-append.queue_time_us":104684}
I20240501 20:01:49.300535 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649b71f9f267437ca51c758dd9ca6987" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:49.302132 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48fd0a8e289401aa7417c42ff2c81b2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:49.302801 29803 raft_consensus.cc:2379] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d in current term 6: Already voted for candidate ee0d97247d394a8facfa774f15f3a67d in this term.
I20240501 20:01:49.303668 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36057b3f24d44b9aac3a1378053a395f" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:49.305663 29801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5525fc4b82a44b0991c3dd6098fde81" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:49.304490 29805 raft_consensus.cc:2454] T 36057b3f24d44b9aac3a1378053a395f P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 3.
I20240501 20:01:49.306717 29800 raft_consensus.cc:1226] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 5. Current term is 6. Ops: []
I20240501 20:01:49.309155 29800 raft_consensus.cc:1261] T a8e9f75387564275a60d38e9a910319c P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 20:01:49.312002 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d86f8848a184a87947fd927c44e5a35" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:49.313905 29802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:49.314033 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211e27a3b0084b89a6dd7550849b5a65" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:49.317688 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d88139f89c47ff84cad674147aae0f" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:49.302748 31467 raft_consensus.cc:504] T 48630ae6eb394ac1b54955dd01cf0ed2 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:49.320698 29606 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 6797d82dda2948d8a7445fcd33e44752, TS fdfeac4190e045f290e003b2298d988d -> TS 888d99522e4844118f8f1cf26f268c8c move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:49.321023 29606 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 6797d82dda2948d8a7445fcd33e44752, TS fdfeac4190e045f290e003b2298d988d -> TS 888d99522e4844118f8f1cf26f268c8c move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:49.321848 31467 leader_election.cc:288] T 48630ae6eb394ac1b54955dd01cf0ed2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:49.292114 31472 consensus_queue.cc:237] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:49.324779 31533 consensus_queue.cc:1035] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.014s
I20240501 20:01:49.328331 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48630ae6eb394ac1b54955dd01cf0ed2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:49.318593 29700 leader_election.cc:302] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:49.332366 31531 raft_consensus.cc:2726] T 77a35893b91247a99912eba787d787cf P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240501 20:01:49.333812 29700 leader_election.cc:302] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:49.335081 31531 raft_consensus.cc:2726] T 3c1a08b455b94ae089396a15c837ddc2 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240501 20:01:49.337895 29700 leader_election.cc:302] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:49.338657 29700 leader_election.cc:302] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:49.339447 31531 raft_consensus.cc:2781] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Leader election won for term 6
I20240501 20:01:49.344329 29700 leader_election.cc:302] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:49.341675 31488 raft_consensus.cc:2781] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:49.346876 31533 raft_consensus.cc:2781] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:49.348275 31087 consensus_queue.cc:1035] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:49.349877 31533 raft_consensus.cc:482] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:49.350239 31533 raft_consensus.cc:3037] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:49.348706 29809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41858 (request call id 1095) took 1252 ms. Trace:
I20240501 20:01:49.350693 29809 rpcz_store.cc:276] 0501 20:01:48.096390 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:48.145300 (+ 48910us) service_pool.cc:225] Handling call
0501 20:01:49.348548 (+1203248us) spinlock_profiling.cc:243] Waited 926 ms on lock 0x7b5400210f54. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787f4fe 00007fe45c7bd9d7 00007fe455ea1c14 00007fe455ea1ba2 00007fe455ea1ad1 00007fe455ea1a4c 00007fe455ea0d02 00007fe455ca2bcc 00007fe455ca2006 00007fe455ca19af
0501 20:01:49.348633 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2130316032}
I20240501 20:01:49.347487 31043 consensus_queue.cc:1046] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.002s
I20240501 20:01:49.349350 31488 raft_consensus.cc:482] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:49.351943 31043 raft_consensus.cc:3032] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [term 5 LEADER]: Stepping down as leader of term 5
I20240501 20:01:49.352255 31488 raft_consensus.cc:3037] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:49.352442 31043 raft_consensus.cc:729] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:49.353281 31043 consensus_queue.cc:260] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:49.354202 31043 raft_consensus.cc:3037] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:49.354490 29657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59136 (request call id 875) took 2721 ms. Trace:
I20240501 20:01:49.354840 29657 rpcz_store.cc:276] 0501 20:01:46.633328 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:46.633443 (+ 115us) service_pool.cc:225] Handling call
0501 20:01:46.651257 (+ 17814us) raft_consensus.cc:1457] Updating replica for 1 ops
0501 20:01:47.018548 (+367291us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:47.018585 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0501 20:01:47.116211 (+ 97626us) log.cc:842] Serialized 26 byte log entry
0501 20:01:47.294333 (+178122us) raft_consensus.cc:1631] Marking committed up to 2
0501 20:01:47.296991 (+ 2658us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:47.297204 (+ 213us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0501 20:01:48.756690 (+1459486us) spinlock_profiling.cc:243] Waited 6.57 ms on lock 0x7b240017ce28. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe454797b51 00007fe4547ac024 00007fe4547aa13b 00007fe454838149 00007fe454837dd5 00007fe454837330 00007fe45752a3a8 00007fe45752ab08 00007fe457502618 00007fe4574fda1a 00007fe4574fce66
0501 20:01:49.354202 (+597512us) raft_consensus.cc:1671] finished
0501 20:01:49.354228 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:49.354438 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":83651,"raft.queue_time_us":103951,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":114,"spinlock_wait_cycles":15120768,"thread_start_us":348825,"threads_started":2,"wal-append.queue_time_us":273704}
I20240501 20:01:49.356627 31533 raft_consensus.cc:504] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:49.347170 31531 raft_consensus.cc:686] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [term 6 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:49.367708 31531 consensus_queue.cc:237] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:49.369184 30417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05df43bdfa479b902b1da297fb5974" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c" is_pre_election: true
I20240501 20:01:49.368288 31472 raft_consensus.cc:482] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:49.371194 31472 raft_consensus.cc:504] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:49.374079 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be383be783234855878cb8fd442c7b05" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:49.375136 31488 raft_consensus.cc:504] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:49.375828 31472 leader_election.cc:288] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:49.379413 29730 raft_consensus.cc:2454] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 888d99522e4844118f8f1cf26f268c8c in term 5.
I20240501 20:01:49.377033 31488 leader_election.cc:288] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:49.377485 31533 leader_election.cc:288] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:49.386561 29734 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "49da4246f5f6438abe0e7e8c9a523d80"
dest_uuid: "fdfeac4190e045f290e003b2298d988d"
mode: GRACEFUL
new_leader_uuid: "ee0d97247d394a8facfa774f15f3a67d"
from {username='slave'} at 127.0.0.1:46074
I20240501 20:01:49.387245 29734 raft_consensus.cc:595] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [term 6 LEADER]: Received request to transfer leadership to TS ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:49.390285 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8787bc4d6b44649b9138d906730110a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:49.390825 29803 raft_consensus.cc:3037] T a8787bc4d6b44649b9138d906730110a P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:49.391121 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36057b3f24d44b9aac3a1378053a395f" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:49.391633 29806 raft_consensus.cc:3037] T 36057b3f24d44b9aac3a1378053a395f P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:49.392611 29772 leader_election.cc:302] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 888d99522e4844118f8f1cf26f268c8c, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:49.400676 29734 raft_consensus.cc:1226] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20240501 20:01:49.402067 29727 raft_consensus.cc:1261] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240501 20:01:49.402364 31472 consensus_queue.cc:1046] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:49.403518 31472 consensus_queue.cc:1035] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:49.406240 31488 raft_consensus.cc:979] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d: : Instructing follower ee0d97247d394a8facfa774f15f3a67d to start an election
I20240501 20:01:49.407012 31531 raft_consensus.cc:1067] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [term 6 LEADER]: Signalling peer ee0d97247d394a8facfa774f15f3a67d to start an election
I20240501 20:01:49.412259 29806 raft_consensus.cc:2454] T 36057b3f24d44b9aac3a1378053a395f P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 4.
I20240501 20:01:49.418735 31472 raft_consensus.cc:2726] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240501 20:01:49.420475 29803 raft_consensus.cc:2454] T a8787bc4d6b44649b9138d906730110a P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 4.
I20240501 20:01:49.421772 31307 raft_consensus.cc:3032] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 5 LEADER]: Stepping down as leader of term 5
I20240501 20:01:49.422263 31307 raft_consensus.cc:729] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:49.423149 31307 consensus_queue.cc:260] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:49.432469 29806 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "49da4246f5f6438abe0e7e8c9a523d80"
dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
from {username='slave'} at 127.0.0.1:41864
I20240501 20:01:49.433534 29806 raft_consensus.cc:482] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:01:49.434120 29806 raft_consensus.cc:3037] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:01:49.436661 31533 raft_consensus.cc:482] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:49.437206 31533 raft_consensus.cc:504] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:49.437696 31043 raft_consensus.cc:482] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:49.438302 30414 raft_consensus.cc:1226] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [term 6 LEADER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 5. Current term is 6. Ops: []
I20240501 20:01:49.438223 31043 raft_consensus.cc:504] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:49.439217 31307 raft_consensus.cc:3037] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:49.439309 31533 leader_election.cc:288] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:49.442128 29700 leader_election.cc:302] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:49.445648 29700 leader_election.cc:302] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:49.448288 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48630ae6eb394ac1b54955dd01cf0ed2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:49.449188 29803 raft_consensus.cc:2454] T 48630ae6eb394ac1b54955dd01cf0ed2 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 2.
I20240501 20:01:49.449846 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcf30cf66714efdac997862dafbe4a2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:49.450453 29805 raft_consensus.cc:2454] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 5.
I20240501 20:01:49.450354 29700 leader_election.cc:302] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:49.451635 29700 leader_election.cc:302] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:49.448102 31531 raft_consensus.cc:2781] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:01:49.443338 31043 leader_election.cc:288] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:49.450758 31488 raft_consensus.cc:2781] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:01:49.452711 31533 raft_consensus.cc:2781] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:49.452989 31488 raft_consensus.cc:686] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:49.453159 31533 raft_consensus.cc:482] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:49.453619 31533 raft_consensus.cc:3037] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Advancing to term 3
W20240501 20:01:49.451890 31440 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.839s user 0.007s sys 0.008s
I20240501 20:01:49.453944 31488 consensus_queue.cc:237] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:49.456738 31440 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.844s user 0.007s sys 0.008s
W20240501 20:01:49.457818 31576 log.cc:924] Time spent T c651044a305c4168805825a53d910899 P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.166s user 0.000s sys 0.001s
I20240501 20:01:49.462281 31525 raft_consensus.cc:2781] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240501 20:01:49.462728 31525 raft_consensus.cc:482] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:49.460529 31531 raft_consensus.cc:686] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:49.463106 31525 raft_consensus.cc:3037] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:49.463702 31531 consensus_queue.cc:237] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:49.464282 31533 raft_consensus.cc:504] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:49.466675 31533 leader_election.cc:288] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:49.467250 29806 raft_consensus.cc:504] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:49.470232 29806 leader_election.cc:288] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 7 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:49.476197 29726 raft_consensus.cc:1261] T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Refusing update from remote peer 888d99522e4844118f8f1cf26f268c8c: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240501 20:01:49.479410 29730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47414 (request call id 84) took 1166 ms. Trace:
I20240501 20:01:49.479779 29730 rpcz_store.cc:276] 0501 20:01:48.313236 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:48.631818 (+318582us) service_pool.cc:225] Handling call
0501 20:01:49.479260 (+847442us) spinlock_profiling.cc:243] Waited 690 ms on lock 0x7b5400153e54. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787f4fe 00007fe45c7bd9d7 00007fe455ea1c14 00007fe455ea1ba2 00007fe455ea1ad1 00007fe455ea1a4c 00007fe455ea0d02 00007fe455ca2bcc 00007fe455ca2006 00007fe455ca19af
0501 20:01:49.479359 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1587156352}
I20240501 20:01:49.483927 30384 leader_election.cc:302] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 888d99522e4844118f8f1cf26f268c8c, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:49.485004 29732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 1073) took 1549 ms. Trace:
I20240501 20:01:49.484973 31455 consensus_queue.cc:1035] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240501 20:01:49.485590 29732 rpcz_store.cc:276] 0501 20:01:47.935466 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:47.936856 (+ 1390us) service_pool.cc:225] Handling call
0501 20:01:49.484820 (+1547964us) spinlock_profiling.cc:243] Waited 711 ms on lock 0x7b54000a7654. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787f4fe 00007fe45c7bd9d7 00007fe455ea1c14 00007fe455ea1ba2 00007fe455ea1ad1 00007fe455ea1a4c 00007fe455ea0d02 00007fe455ca2bcc 00007fe455ca2006 00007fe455ca19af
0501 20:01:49.484950 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1635228928}
I20240501 20:01:49.486744 31455 raft_consensus.cc:2781] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:01:49.487358 31455 raft_consensus.cc:686] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [term 5 LEADER]: Becoming Leader. State: Replica: 888d99522e4844118f8f1cf26f268c8c, State: Running, Role: LEADER
I20240501 20:01:49.491331 31455 consensus_queue.cc:237] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:49.491869 29732 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "668b4a4d782c4278932dd8413dc3ab3e"
dest_uuid: "fdfeac4190e045f290e003b2298d988d"
mode: GRACEFUL
new_leader_uuid: "ee0d97247d394a8facfa774f15f3a67d"
from {username='slave'} at 127.0.0.1:46074
I20240501 20:01:49.492489 29732 raft_consensus.cc:595] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Received request to transfer leadership to TS ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:49.492820 29732 raft_consensus.cc:603] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240501 20:01:49.494148 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49da4246f5f6438abe0e7e8c9a523d80" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: true dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:49.494704 29732 raft_consensus.cc:3032] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [term 6 LEADER]: Stepping down as leader of term 6
I20240501 20:01:49.494899 29732 raft_consensus.cc:729] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:49.495497 29804 raft_consensus.cc:1261] T 696121417ea9403990456cfc6fc4f416 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Refusing update from remote peer 888d99522e4844118f8f1cf26f268c8c: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240501 20:01:49.495419 29732 consensus_queue.cc:260] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:49.505456 29732 raft_consensus.cc:3037] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:01:49.504527 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48630ae6eb394ac1b54955dd01cf0ed2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:49.512845 29806 raft_consensus.cc:3037] T 48630ae6eb394ac1b54955dd01cf0ed2 P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:49.515149 29732 raft_consensus.cc:2454] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 7.
I20240501 20:01:49.515255 29657 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a5525fc4b82a44b0991c3dd6098fde81"
dest_uuid: "4c8da1d580614449a636b855bf11b95f"
mode: GRACEFUL
new_leader_uuid: "ee0d97247d394a8facfa774f15f3a67d"
from {username='slave'} at 127.0.0.1:60736
I20240501 20:01:49.515792 29657 raft_consensus.cc:595] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Received request to transfer leadership to TS ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:49.516120 29657 raft_consensus.cc:603] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240501 20:01:49.516430 31507 consensus_queue.cc:1035] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:01:49.517418 31525 raft_consensus.cc:504] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:49.519311 31525 leader_election.cc:288] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:49.519505 29732 raft_consensus.cc:1261] T 696121417ea9403990456cfc6fc4f416 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Refusing update from remote peer 888d99522e4844118f8f1cf26f268c8c: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240501 20:01:49.519932 29726 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ec073fe645614fe28e7837de3c22510b"
dest_uuid: "fdfeac4190e045f290e003b2298d988d"
mode: GRACEFUL
new_leader_uuid: "4c8da1d580614449a636b855bf11b95f"
from {username='slave'} at 127.0.0.1:46074
I20240501 20:01:49.520476 29726 raft_consensus.cc:595] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Received request to transfer leadership to TS 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:49.520825 29726 raft_consensus.cc:603] T ec073fe645614fe28e7837de3c22510b P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240501 20:01:49.522539 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcf30cf66714efdac997862dafbe4a2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:49.523242 29804 raft_consensus.cc:3037] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:49.528661 31533 raft_consensus.cc:482] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:49.529007 29731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36270 (request call id 1079) took 1214 ms. Trace:
I20240501 20:01:49.529295 31533 raft_consensus.cc:504] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:49.529536 29731 rpcz_store.cc:276] 0501 20:01:48.333189 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:48.681021 (+347832us) service_pool.cc:225] Handling call
0501 20:01:49.528810 (+847789us) spinlock_profiling.cc:243] Waited 243 ms on lock 0x7b5400290254. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787f4fe 00007fe45c7bd9d7 00007fe455ea1c14 00007fe455ea1ba2 00007fe455ea1ad1 00007fe455ea1a4c 00007fe455ea0d02 00007fe455ca2bcc 00007fe455ca2006 00007fe455ca19af
0501 20:01:49.528949 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":559250176}
I20240501 20:01:49.531507 29807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41864 (request call id 1154) took 1644 ms. Trace:
I20240501 20:01:49.531972 29807 rpcz_store.cc:276] 0501 20:01:47.886883 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:48.469796 (+582913us) service_pool.cc:225] Handling call
0501 20:01:48.499388 (+ 29592us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:48.969348 (+469960us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:48.969379 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0501 20:01:48.969421 (+ 42us) raft_consensus.cc:1631] Marking committed up to 3
0501 20:01:48.969452 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:49.531268 (+561816us) spinlock_profiling.cc:243] Waited 470 ms on lock 0x7b54002ef754. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787c1a1 00007fe45787abee 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc 00007fe455e9e782 00007fe455ca2bcc 00007fe455ca2006
0501 20:01:49.531321 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:49.531448 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1080327552}
W20240501 20:01:49.537065 31587 log.cc:924] Time spent T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
I20240501 20:01:49.538479 31536 raft_consensus.cc:482] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:49.539094 31536 raft_consensus.cc:504] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:49.540830 29773 leader_election.cc:302] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:49.541473 31536 leader_election.cc:288] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:49.542412 31501 raft_consensus.cc:2781] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [term 7 FOLLOWER]: Leader election won for term 7
I20240501 20:01:49.542912 31501 raft_consensus.cc:686] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [term 7 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:49.543735 31501 consensus_queue.cc:237] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:49.531883 31533 leader_election.cc:288] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:49.547574 29806 raft_consensus.cc:2454] T 48630ae6eb394ac1b54955dd01cf0ed2 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 3.
I20240501 20:01:49.557067 31444 consensus_queue.cc:1035] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20240501 20:01:49.562692 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77be7101c0fe4085b6706296f609870a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:49.562980 29802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9472c53f79ec423f82c396bd980674e2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:49.564122 29802 raft_consensus.cc:2396] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20240501 20:01:49.574622 29805 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc"
dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
mode: GRACEFUL
new_leader_uuid: "4c8da1d580614449a636b855bf11b95f"
from {username='slave'} at 127.0.0.1:33582
I20240501 20:01:49.566268 29700 leader_election.cc:302] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:49.571242 29804 raft_consensus.cc:2454] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 6.
I20240501 20:01:49.562682 31525 raft_consensus.cc:482] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:49.600950 29700 leader_election.cc:302] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:49.605688 29735 raft_consensus.cc:1261] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240501 20:01:49.608469 29805 raft_consensus.cc:595] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 5 LEADER]: Received request to transfer leadership to TS 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:49.602126 31531 raft_consensus.cc:2781] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:01:49.609735 31536 raft_consensus.cc:2781] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Leader election won for term 6
I20240501 20:01:49.594457 31525 raft_consensus.cc:504] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } }
I20240501 20:01:49.638119 28519 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240501 20:01:49.619892 31470 consensus_queue.cc:1035] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.013s
I20240501 20:01:49.628809 31536 raft_consensus.cc:686] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 6 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:49.625666 31531 raft_consensus.cc:686] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d [term 3 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:49.644038 29804 raft_consensus.cc:1261] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Refusing update from remote peer 888d99522e4844118f8f1cf26f268c8c: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20240501 20:01:49.642160 31588 log.cc:924] Time spent T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.093s user 0.000s sys 0.001s
W20240501 20:01:49.646404 31586 log.cc:924] Time spent T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d: Append to log took a long time: real 0.084s user 0.001s sys 0.000s
I20240501 20:01:49.652685 29557 catalog_manager.cc:5484] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:49.654779 31426 raft_consensus.cc:482] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:49.655324 31426 raft_consensus.cc:504] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:49.696278 31536 consensus_queue.cc:237] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:49.732964 28519 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:01:49.707010 31531 consensus_queue.cc:237] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:49.726642 30383 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.27.217.193:33959 timed out after 1.704s (SENT) [suppressed 105 similar messages]
I20240501 20:01:49.732781 31498 raft_consensus.cc:482] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:49.739409 31498 raft_consensus.cc:504] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:49.740547 31554 log.cc:924] Time spent T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c: Append to log took a long time: real 0.079s user 0.000s sys 0.010s
I20240501 20:01:49.740986 30944 consensus_queue.cc:1035] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:01:49.741700 31498 leader_election.cc:288] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:49.732757 31440 raft_consensus.cc:465] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [term 2 LEADER]: Not starting pre-election -- already a leader
I20240501 20:01:49.748376 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8edcd903224d48f889b46e1b5318e0ec" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:49.748736 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69915dd5992641e3a24f7f45f1cd5c0c" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:49.734061 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:01:49.749358 29804 raft_consensus.cc:2454] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 5.
I20240501 20:01:49.753224 29731 raft_consensus.cc:1261] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 7 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20240501 20:01:49.737668 31593 log.cc:924] Time spent T cb05df43bdfa479b902b1da297fb5974 P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.075s user 0.003s sys 0.000s
I20240501 20:01:49.757144 29700 leader_election.cc:302] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:49.758354 31533 raft_consensus.cc:2781] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240501 20:01:49.758733 31533 raft_consensus.cc:482] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:49.759105 31533 raft_consensus.cc:3037] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:49.753217 31525 leader_election.cc:288] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861), 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959)
I20240501 20:01:49.753933 31426 leader_election.cc:288] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:49.799094 31533 raft_consensus.cc:504] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } }
I20240501 20:01:49.805678 31470 consensus_queue.cc:1035] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:49.806285 29809 raft_consensus.cc:1261] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20240501 20:01:49.808771 29735 raft_consensus.cc:1261] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240501 20:01:49.762881 31307 raft_consensus.cc:482] T 84fa28fb55f44b61ba1a88ed775a78ba P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
W20240501 20:01:49.887967 31592 log.cc:924] Time spent T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c: Append to log took a long time: real 0.161s user 0.000s sys 0.001s
W20240501 20:01:49.844230 30383 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 117614us
I20240501 20:01:49.892069 31501 consensus_queue.cc:1035] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.016s
I20240501 20:01:49.917831 29558 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:43370 (request call id 155) took 4147 ms. Trace:
I20240501 20:01:49.889261 31488 consensus_queue.cc:1035] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
I20240501 20:01:49.890900 31307 raft_consensus.cc:504] T 84fa28fb55f44b61ba1a88ed775a78ba P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:49.918282 29558 rpcz_store.cc:276] 0501 20:01:45.758998 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:45.759238 (+ 240us) service_pool.cc:225] Handling call
0501 20:01:49.906418 (+4147180us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0501 20:01:46.784958 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 20:01:46.909027 (+124069us) write_op.cc:424] Acquiring schema lock in shared mode
0501 20:01:46.920951 (+ 11924us) write_op.cc:427] Acquired schema lock
0501 20:01:46.924459 (+ 3508us) tablet.cc:660] Decoding operations
0501 20:01:47.213004 (+288545us) write_op.cc:612] Acquiring the partition lock for write op
0501 20:01:47.213082 (+ 78us) write_op.cc:633] Partition lock acquired for write op
0501 20:01:47.213111 (+ 29us) tablet.cc:683] Acquiring locks for 15 operations
0501 20:01:47.265573 (+ 52462us) tablet.cc:699] Row locks acquired
0501 20:01:47.265603 (+ 30us) write_op.cc:260] PREPARE: finished
0501 20:01:47.265780 (+ 177us) write_op.cc:270] Start()
0501 20:01:47.265916 (+ 136us) write_op.cc:276] Timestamp: P: 1714593707265733 usec, L: 0
0501 20:01:47.265944 (+ 28us) op_driver.cc:331] REPLICATION: starting
0501 20:01:47.266692 (+ 748us) log.cc:842] Serialized 6073 byte log entry
0501 20:01:47.478253 (+211561us) op_driver.cc:444] REPLICATION: finished
0501 20:01:47.783239 (+304986us) write_op.cc:301] APPLY: starting
0501 20:01:47.788615 (+ 5376us) tablet.cc:1365] starting BulkCheckPresence
0501 20:01:47.863909 (+ 75294us) tablet.cc:1368] finished BulkCheckPresence
0501 20:01:47.876706 (+ 12797us) tablet.cc:1370] starting ApplyRowOperation cycle
0501 20:01:47.909931 (+ 33225us) tablet.cc:1381] finished ApplyRowOperation cycle
0501 20:01:47.910159 (+ 228us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=15
0501 20:01:47.910201 (+ 42us) write_op.cc:312] APPLY: finished
0501 20:01:47.910509 (+ 308us) log.cc:842] Serialized 106 byte log entry
0501 20:01:47.910672 (+ 163us) write_op.cc:481] Releasing partition, row and schema locks
0501 20:01:47.932316 (+ 21644us) write_op.cc:446] Released schema lock
0501 20:01:47.932374 (+ 58us) write_op.cc:341] FINISH: Updating metrics
0501 20:01:47.932628 (+ 254us) write_op.cc:481] Releasing partition, row and schema locks
0501 20:01:47.932655 (+ 27us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":91243,"reported_tablets":41,"child_traces":[["op",{"apply.queue_time_us":276917,"apply.run_cpu_time_us":4829,"apply.run_wall_time_us":166631,"num_ops":15,"prepare.queue_time_us":1765,"prepare.run_cpu_time_us":7486,"prepare.run_wall_time_us":486082,"replication_time_us":200417,"thread_start_us":163381,"threads_started":1}]]}
I20240501 20:01:49.936604 31434 raft_consensus.cc:482] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240501 20:01:49.941627 29830 debug-util.cc:398] Leaking SignalData structure 0x7b0800151300 after lost signal to thread 28522
I20240501 20:01:49.937063 31434 raft_consensus.cc:504] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:49.965606 29830 debug-util.cc:398] Leaking SignalData structure 0x7b0800113ee0 after lost signal to thread 29592
W20240501 20:01:49.966351 29830 debug-util.cc:398] Leaking SignalData structure 0x7b08001e1c60 after lost signal to thread 29623
W20240501 20:01:49.966718 29830 debug-util.cc:398] Leaking SignalData structure 0x7b08001deb40 after lost signal to thread 29625
I20240501 20:01:49.967095 31434 leader_election.cc:288] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:49.968106 29830 debug-util.cc:398] Leaking SignalData structure 0x7b08003abc20 after lost signal to thread 29684
W20240501 20:01:49.968552 29830 debug-util.cc:398] Leaking SignalData structure 0x7b08003b7ba0 after lost signal to thread 29685
I20240501 20:01:49.970088 31307 leader_election.cc:288] T 84fa28fb55f44b61ba1a88ed775a78ba P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:49.970305 31533 leader_election.cc:288] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 election: Requested vote from peers ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861), 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959)
W20240501 20:01:49.970633 29830 debug-util.cc:398] Leaking SignalData structure 0x7b0800202bc0 after lost signal to thread 29759
W20240501 20:01:49.971051 29830 debug-util.cc:398] Leaking SignalData structure 0x7b0800305580 after lost signal to thread 29760
I20240501 20:01:49.975662 29726 raft_consensus.cc:1261] T 25e07220db7444ae97f294d70eda6a93 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Refusing update from remote peer 888d99522e4844118f8f1cf26f268c8c: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20240501 20:01:49.972535 29830 debug-util.cc:398] Leaking SignalData structure 0x7b0800380e20 after lost signal to thread 29833
I20240501 20:01:49.978394 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69915dd5992641e3a24f7f45f1cd5c0c" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:49.978983 29807 raft_consensus.cc:3037] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Advancing to term 6
W20240501 20:01:49.979795 29830 debug-util.cc:398] Leaking SignalData structure 0x7b08002f3f40 after lost signal to thread 30445
W20240501 20:01:49.980216 29830 debug-util.cc:398] Leaking SignalData structure 0x7b08002602c0 after lost signal to thread 30446
W20240501 20:01:49.986604 31604 log.cc:924] Time spent T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20240501 20:01:49.988517 30944 consensus_queue.cc:1035] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 20:01:49.988884 29830 debug-util.cc:398] Leaking SignalData structure 0x7b08003858e0 after lost signal to thread 31466
W20240501 20:01:49.989392 29830 debug-util.cc:398] Leaking SignalData structure 0x7b08003863a0 after lost signal to thread 31471
W20240501 20:01:49.990176 29830 debug-util.cc:398] Leaking SignalData structure 0x7b08003853a0 after lost signal to thread 31510
W20240501 20:01:49.990515 29830 debug-util.cc:398] Leaking SignalData structure 0x7b08003855e0 after lost signal to thread 31511
I20240501 20:01:49.993029 29807 raft_consensus.cc:2454] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 6.
I20240501 20:01:49.994796 29700 leader_election.cc:302] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
W20240501 20:01:49.995759 29830 debug-util.cc:398] Leaking SignalData structure 0x7b08003855c0 after lost signal to thread 31512
I20240501 20:01:50.007611 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84fa28fb55f44b61ba1a88ed775a78ba" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:50.013051 29734 raft_consensus.cc:1261] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240501 20:01:50.012284 31518 raft_consensus.cc:2781] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Leader election won for term 6
I20240501 20:01:50.017417 31448 raft_consensus.cc:482] T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:50.015300 31425 raft_consensus.cc:482] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:50.021474 31448 raft_consensus.cc:504] T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:50.021627 31425 raft_consensus.cc:504] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:50.023581 31448 leader_election.cc:288] T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:50.023797 31425 leader_election.cc:288] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:50.016840 31553 raft_consensus.cc:482] T 1ca3948261374cef8b1b206a43c00173 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:50.029966 29807 raft_consensus.cc:1261] T a8787bc4d6b44649b9138d906730110a P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240501 20:01:50.031673 31553 raft_consensus.cc:504] T 1ca3948261374cef8b1b206a43c00173 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:50.027096 31518 raft_consensus.cc:686] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [term 6 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:50.020606 31467 consensus_queue.cc:1035] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.020s
I20240501 20:01:50.050529 31553 leader_election.cc:288] T 1ca3948261374cef8b1b206a43c00173 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:50.053009 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 859375us
I20240501 20:01:50.053400 31518 consensus_queue.cc:237] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } }
I20240501 20:01:50.054152 29624 leader_election.cc:302] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:50.040984 31531 consensus_queue.cc:1035] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.019s
I20240501 20:01:50.082643 31413 raft_consensus.cc:2726] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240501 20:01:50.101692 31607 raft_consensus.cc:482] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:50.102479 31607 raft_consensus.cc:504] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:50.105079 31607 leader_election.cc:288] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:50.106223 29809 raft_consensus.cc:1261] T 36057b3f24d44b9aac3a1378053a395f P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240501 20:01:50.107218 31432 raft_consensus.cc:482] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:50.107335 31472 raft_consensus.cc:482] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:50.107857 31432 raft_consensus.cc:504] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:50.108124 31472 raft_consensus.cc:504] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:50.110102 31432 leader_election.cc:288] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:50.110399 31472 leader_election.cc:288] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:50.111068 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649b71f9f267437ca51c758dd9ca6987" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:50.115619 31063 raft_consensus.cc:482] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:50.116125 31063 raft_consensus.cc:504] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:50.116849 31448 raft_consensus.cc:1050] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f: attempting to promote NON_VOTER 888d99522e4844118f8f1cf26f268c8c to VOTER
W20240501 20:01:50.117211 31448 raft_consensus.cc:1054] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f: Unable to promote non-voter 888d99522e4844118f8f1cf26f268c8c: Illegal state: Leader has not yet committed an operation in its own term
I20240501 20:01:50.117854 31063 leader_election.cc:288] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:50.119014 29657 raft_consensus.cc:1261] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 4c8da1d580614449a636b855bf11b95f [term 7 FOLLOWER]: Refusing update from remote peer 888d99522e4844118f8f1cf26f268c8c: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20240501 20:01:50.121270 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25e07220db7444ae97f294d70eda6a93" candidate_uuid: "888d99522e4844118f8f1cf26f268c8c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:50.121980 29652 raft_consensus.cc:2362] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 888d99522e4844118f8f1cf26f268c8c in term 5. Re-sending same reply.
I20240501 20:01:50.123380 29660 raft_consensus.cc:1261] T 25e07220db7444ae97f294d70eda6a93 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Refusing update from remote peer 888d99522e4844118f8f1cf26f268c8c: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20240501 20:01:50.124722 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 60806.1us
I20240501 20:01:50.125396 29624 leader_election.cc:302] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:50.126902 29698 leader_election.cc:302] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:50.129014 29734 raft_consensus.cc:1261] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d [term 7 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240501 20:01:50.129423 29698 leader_election.cc:302] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:50.130650 29624 leader_election.cc:302] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:50.131873 29698 leader_election.cc:302] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:50.134598 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5345f6eb62449ba69b4a9c2e349b31" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:50.136337 29804 raft_consensus.cc:1226] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20240501 20:01:50.136575 29806 raft_consensus.cc:1226] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [term 4 LEADER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 3. Current term is 4. Ops: []
I20240501 20:01:50.137488 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49da4246f5f6438abe0e7e8c9a523d80" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:50.137485 29809 raft_consensus.cc:1226] T 48630ae6eb394ac1b54955dd01cf0ed2 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 2. Current term is 3. Ops: []
I20240501 20:01:50.138273 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8cd40cb4044d18a86001ce6ddf97ab" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:50.093008 31333 raft_consensus.cc:482] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:50.143687 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77be7101c0fe4085b6706296f609870a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:50.146075 29731 raft_consensus.cc:2454] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 4.
I20240501 20:01:50.093055 31426 raft_consensus.cc:482] T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:50.144798 29802 raft_consensus.cc:1226] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 5. Current term is 6. Ops: []
I20240501 20:01:50.145202 29803 raft_consensus.cc:1226] T 36057b3f24d44b9aac3a1378053a395f P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 3. Current term is 4. Ops: []
I20240501 20:01:50.144783 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e9f75387564275a60d38e9a910319c" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:50.148741 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31fff64b453a423eb0f567b3171564fb" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:50.150404 29625 connection.cc:232] RPC call timeout handler was delayed by -1.3863984080001046s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21640000000000004s for a response
I20240501 20:01:50.151449 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8edcd903224d48f889b46e1b5318e0ec" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:50.130594 31518 raft_consensus.cc:2726] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240501 20:01:50.151890 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c597f15e844f30837012baed4b0501" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:50.139396 31440 raft_consensus.cc:2726] T 7a3f29f0eeea4b119984634202e5d9cd P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240501 20:01:50.149845 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c651044a305c4168805825a53d910899" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:50.154738 29652 raft_consensus.cc:3037] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Advancing to term 6
I20240501 20:01:50.152526 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8787bc4d6b44649b9138d906730110a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:50.141695 31449 raft_consensus.cc:2781] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240501 20:01:50.155635 29773 leader_election.cc:302] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:50.155725 29624 leader_election.cc:302] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:50.156334 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48fd0a8e289401aa7417c42ff2c81b2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:50.158010 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e49dd5180f04313ab33942fb3ddc147" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:50.159440 29808 raft_consensus.cc:1261] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20240501 20:01:50.170809 29655 raft_consensus.cc:3037] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:50.170958 29808 pending_rounds.cc:76] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d: Aborting all ops after (but not including) 2
I20240501 20:01:50.149688 31087 raft_consensus.cc:2726] T 6797d82dda2948d8a7445fcd33e44752 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240501 20:01:50.175063 29624 leader_election.cc:302] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:50.171335 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca3948261374cef8b1b206a43c00173" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:50.171969 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5525fc4b82a44b0991c3dd6098fde81" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:50.155445 31506 raft_consensus.cc:482] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:50.151391 31421 raft_consensus.cc:2781] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:01:50.175853 29653 raft_consensus.cc:2454] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 5.
I20240501 20:01:50.179812 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07bdb8b09b8c4ef4b8e49daa9b8c01d6" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:50.180769 29656 raft_consensus.cc:2379] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d in current term 6: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:50.139474 31605 consensus_queue.cc:1035] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
I20240501 20:01:50.152815 31427 raft_consensus.cc:482] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:50.171342 29808 pending_rounds.cc:97] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20240501 20:01:50.157804 29654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece8d4ed84c34b748f7b6272b31c90ac" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:50.152249 31426 raft_consensus.cc:504] T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:50.159078 31449 raft_consensus.cc:482] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting leader election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:50.149814 31333 raft_consensus.cc:504] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:50.159672 31460 raft_consensus.cc:482] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:50.204103 29808 raft_consensus.cc:2866] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 20:01:50.206116 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:50.206487 31434 raft_consensus.cc:2781] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240501 20:01:50.177119 31498 raft_consensus.cc:2781] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240501 20:01:50.159248 31501 consensus_queue.cc:1035] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.041s
I20240501 20:01:50.212935 29657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649b71f9f267437ca51c758dd9ca6987" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:50.212985 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0374bb790ed345ed90452595f71383a2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:50.172900 31444 consensus_queue.cc:1035] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.031s
I20240501 20:01:50.217538 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211e27a3b0084b89a6dd7550849b5a65" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:50.219213 29658 raft_consensus.cc:3037] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:50.219671 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d86f8848a184a87947fd927c44e5a35" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:50.199003 31421 raft_consensus.cc:686] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [term 5 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:50.207456 31510 consensus_queue.cc:1046] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
I20240501 20:01:50.228119 31333 leader_election.cc:288] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:50.218406 31498 raft_consensus.cc:482] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:50.212469 31449 raft_consensus.cc:3037] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:50.183774 31506 raft_consensus.cc:504] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:50.242842 29801 raft_consensus.cc:1226] T a8787bc4d6b44649b9138d906730110a P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 3. Current term is 4. Ops: []
I20240501 20:01:50.217795 31434 raft_consensus.cc:482] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting leader election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:50.244685 29658 raft_consensus.cc:2454] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 4.
I20240501 20:01:50.216295 29654 raft_consensus.cc:2454] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 5.
I20240501 20:01:50.247818 31333 raft_consensus.cc:3032] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 5 LEADER]: Stepping down as leader of term 5
W20240501 20:01:50.251667 31616 log.cc:924] Time spent T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.094s user 0.000s sys 0.000s
I20240501 20:01:50.251374 31421 consensus_queue.cc:237] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:50.251960 31333 raft_consensus.cc:729] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:50.253491 31426 leader_election.cc:288] T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:50.252871 31333 consensus_queue.cc:260] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:50.239910 31511 raft_consensus.cc:482] T add9d7579351483fb268f89a44e95d8a P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 888d99522e4844118f8f1cf26f268c8c)
I20240501 20:01:50.209444 31427 raft_consensus.cc:504] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:50.257205 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36057b3f24d44b9aac3a1378053a395f" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:50.238430 31498 raft_consensus.cc:3037] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Advancing to term 5
W20240501 20:01:50.258678 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 60822.9us
I20240501 20:01:50.261546 31333 raft_consensus.cc:3037] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:50.218030 31460 raft_consensus.cc:504] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:50.222363 31512 consensus_queue.cc:1046] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
I20240501 20:01:50.263890 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d88139f89c47ff84cad674147aae0f" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
W20240501 20:01:50.267423 30383 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 95674.2us
I20240501 20:01:50.250504 31434 raft_consensus.cc:3037] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:50.273171 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36057b3f24d44b9aac3a1378053a395f" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:50.263489 31506 leader_election.cc:288] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:50.278301 29734 raft_consensus.cc:1226] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [term 6 LEADER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 5. Current term is 6. Ops: []
I20240501 20:01:50.280198 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48630ae6eb394ac1b54955dd01cf0ed2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:50.282858 29731 raft_consensus.cc:1226] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d [term 3 LEADER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 2. Current term is 3. Ops: []
I20240501 20:01:50.283217 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49da4246f5f6438abe0e7e8c9a523d80" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:50.286365 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcf30cf66714efdac997862dafbe4a2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:50.287711 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e9f75387564275a60d38e9a910319c" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:50.289525 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31fff64b453a423eb0f567b3171564fb" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:50.292182 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca3948261374cef8b1b206a43c00173" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:50.293116 29727 raft_consensus.cc:1261] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240501 20:01:50.276981 29651 raft_consensus.cc:1261] T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 20:01:50.293653 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5525fc4b82a44b0991c3dd6098fde81" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:50.270221 31425 consensus_queue.cc:1046] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.016s
I20240501 20:01:50.264355 31511 raft_consensus.cc:504] T add9d7579351483fb268f89a44e95d8a P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:50.295667 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbcf30cf66714efdac997862dafbe4a2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:50.301676 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48630ae6eb394ac1b54955dd01cf0ed2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:50.296288 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8cd40cb4044d18a86001ce6ddf97ab" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:50.300765 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfac893b4b4f457694a4eb184b6abb53" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:50.289289 31510 raft_consensus.cc:3032] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 6 LEADER]: Stepping down as leader of term 6
I20240501 20:01:50.311746 29655 raft_consensus.cc:2454] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 6.
I20240501 20:01:50.315804 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b337b8e3e38d4ad6b1076fbd0d0a1609" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:50.324604 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f5d1686214485d91a0baf4b870740e" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:50.327153 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69915dd5992641e3a24f7f45f1cd5c0c" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:50.327692 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77be7101c0fe4085b6706296f609870a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:50.328600 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb27193abbf4644a48662909bc7b04a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:50.337266 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69915dd5992641e3a24f7f45f1cd5c0c" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:50.295382 29654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8787bc4d6b44649b9138d906730110a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:50.349475 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "456035a1cb3f4f7da52d0fc1dd1c656c" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:50.350880 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48630ae6eb394ac1b54955dd01cf0ed2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:50.354009 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84fa28fb55f44b61ba1a88ed775a78ba" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:50.355010 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c651044a305c4168805825a53d910899" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:50.319499 31510 raft_consensus.cc:729] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:50.355880 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8edcd903224d48f889b46e1b5318e0ec" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:50.357475 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9472c53f79ec423f82c396bd980674e2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:50.357753 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af08384057a4d85977481472f460168" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:50.358548 29658 raft_consensus.cc:2396] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20240501 20:01:50.359862 29652 raft_consensus.cc:2454] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 6.
I20240501 20:01:50.365670 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5525fc4b82a44b0991c3dd6098fde81" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:50.384683 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0374bb790ed345ed90452595f71383a2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:50.387383 29656 raft_consensus.cc:2379] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 5: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
W20240501 20:01:50.363080 31613 log.cc:924] Time spent T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.154s user 0.000s sys 0.003s
I20240501 20:01:50.403319 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c5345f6eb62449ba69b4a9c2e349b31" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:50.404490 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77be7101c0fe4085b6706296f609870a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:50.404934 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a2bba62f434d1c8489b55c8fec2187" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:50.365088 31432 raft_consensus.cc:2741] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20240501 20:01:50.416028 29731 raft_consensus.cc:1226] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [term 6 LEADER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 5. Current term is 6. Ops: []
I20240501 20:01:50.412187 29653 raft_consensus.cc:3032] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:01:50.417152 29653 raft_consensus.cc:729] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:50.410133 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af08384057a4d85977481472f460168" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:50.365496 29651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:50.418032 29653 consensus_queue.cc:260] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:50.361246 31063 consensus_queue.cc:1046] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.049s
I20240501 20:01:50.431903 29653 raft_consensus.cc:3037] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:50.431547 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be383be783234855878cb8fd442c7b05" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:50.436610 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49da4246f5f6438abe0e7e8c9a523d80" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: true dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:50.408627 31472 raft_consensus.cc:482] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:50.341693 31620 consensus_queue.cc:1035] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.078s
I20240501 20:01:50.387248 31425 consensus_queue.cc:1035] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.045s
I20240501 20:01:50.384265 31448 consensus_queue.cc:1046] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.046s
I20240501 20:01:50.420836 31467 raft_consensus.cc:482] T 010ceabf3e5a4b1c9330b80dade765bb P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:50.456751 31448 consensus_queue.cc:1225] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer fdfeac4190e045f290e003b2298d988d log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20240501 20:01:50.457444 31448 raft_consensus.cc:3032] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:01:50.457809 31448 raft_consensus.cc:729] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:50.458746 31448 consensus_queue.cc:260] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:50.446170 31427 leader_election.cc:288] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:50.461082 29625 leader_election.cc:302] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:50.462841 31427 raft_consensus.cc:2726] T 458740ab5c444b6ab3d1991c69db5084 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240501 20:01:50.466104 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e49dd5180f04313ab33942fb3ddc147" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:50.466729 29655 raft_consensus.cc:2362] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 6. Re-sending same reply.
I20240501 20:01:50.469455 29625 leader_election.cc:302] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters: fdfeac4190e045f290e003b2298d988d
I20240501 20:01:50.444068 31063 consensus_queue.cc:1225] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer fdfeac4190e045f290e003b2298d988d log is divergent from this leader: its last log entry 6.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240501 20:01:50.433467 31421 consensus_queue.cc:1046] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.037s
I20240501 20:01:50.477813 29653 raft_consensus.cc:1261] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240501 20:01:50.479199 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8787bc4d6b44649b9138d906730110a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:50.480022 29653 raft_consensus.cc:2454] T a8787bc4d6b44649b9138d906730110a P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 4.
I20240501 20:01:50.448024 31500 raft_consensus.cc:482] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:50.471601 29654 raft_consensus.cc:1261] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20240501 20:01:50.490367 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac2171683884f6f97cdfeb0a656cb49" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:50.459960 31448 raft_consensus.cc:3037] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:50.446202 31472 raft_consensus.cc:504] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:50.511559 29625 leader_election.cc:302] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters: fdfeac4190e045f290e003b2298d988d
I20240501 20:01:50.489142 31063 raft_consensus.cc:2781] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 20:01:50.451618 31467 raft_consensus.cc:504] T 010ceabf3e5a4b1c9330b80dade765bb P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:50.545859 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 261091us
I20240501 20:01:50.448446 31510 consensus_queue.cc:260] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:50.549794 29655 raft_consensus.cc:3032] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:01:50.519662 31421 consensus_queue.cc:1225] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer fdfeac4190e045f290e003b2298d988d log is divergent from this leader: its last log entry 6.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240501 20:01:50.493597 31511 leader_election.cc:288] T add9d7579351483fb268f89a44e95d8a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:50.522317 31517 raft_consensus.cc:2781] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Leader election won for term 6
I20240501 20:01:50.520826 31634 raft_consensus.cc:482] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:50.521574 31063 raft_consensus.cc:482] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting leader election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:50.501555 31500 raft_consensus.cc:504] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:50.499258 31460 leader_election.cc:288] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:50.550994 29655 raft_consensus.cc:729] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:50.532528 30414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add9d7579351483fb268f89a44e95d8a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c" is_pre_election: true
I20240501 20:01:50.559331 31063 raft_consensus.cc:3037] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:50.549446 31470 raft_consensus.cc:482] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
W20240501 20:01:50.571231 31627 log.cc:924] Time spent T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.063s user 0.001s sys 0.000s
W20240501 20:01:50.555172 31258 consensus_queue.cc:1159] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ee0d97247d394a8facfa774f15f3a67d" responder_term: 6 status { last_received { term: 6 index: 3 } last_committed_idx: 2 last_received_current_leader { term: 6 index: 3 } }
I20240501 20:01:50.550451 31425 consensus_queue.cc:1046] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.027s
I20240501 20:01:50.584375 31470 raft_consensus.cc:504] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:50.588137 29654 raft_consensus.cc:1261] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20240501 20:01:50.592587 29772 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 303488us
I20240501 20:01:50.571630 31517 raft_consensus.cc:686] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 6 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
W20240501 20:01:50.600102 29625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 81872.9us
I20240501 20:01:50.579049 29655 consensus_queue.cc:260] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:50.601169 29655 raft_consensus.cc:3037] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:50.603595 29625 leader_election.cc:302] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters: fdfeac4190e045f290e003b2298d988d
I20240501 20:01:50.604490 29772 leader_election.cc:302] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters: fdfeac4190e045f290e003b2298d988d
I20240501 20:01:50.608423 31433 raft_consensus.cc:2741] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20240501 20:01:50.559641 31634 raft_consensus.cc:504] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:50.618510 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 43995.4us
I20240501 20:01:50.620513 29625 leader_election.cc:302] T 3c5345f6eb62449ba69b4a9c2e349b31 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:50.610569 29654 raft_consensus.cc:3037] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:01:50.626902 29625 leader_election.cc:302] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:50.608932 31639 raft_consensus.cc:2781] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240501 20:01:50.629890 31639 raft_consensus.cc:482] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:50.630259 31639 raft_consensus.cc:3037] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:50.630488 31553 raft_consensus.cc:3032] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [term 5 LEADER]: Stepping down as leader of term 5
I20240501 20:01:50.630810 31553 raft_consensus.cc:729] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:50.610409 31517 consensus_queue.cc:237] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:50.632370 31646 consensus_queue.cc:1035] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:50.634287 31500 leader_election.cc:288] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:50.634760 31500 raft_consensus.cc:979] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d: : Instructing follower 4c8da1d580614449a636b855bf11b95f to start an election
I20240501 20:01:50.635171 31500 raft_consensus.cc:1067] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 5 LEADER]: Signalling peer 4c8da1d580614449a636b855bf11b95f to start an election
I20240501 20:01:50.598879 31449 raft_consensus.cc:504] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:50.629418 31632 log.cc:924] Time spent T 36057b3f24d44b9aac3a1378053a395f P ee0d97247d394a8facfa774f15f3a67d: Append to log took a long time: real 0.083s user 0.001s sys 0.000s
I20240501 20:01:50.639878 31506 raft_consensus.cc:3032] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [term 5 LEADER]: Stepping down as leader of term 5
I20240501 20:01:50.640295 31506 raft_consensus.cc:729] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:50.622375 31433 raft_consensus.cc:2726] T 3c5345f6eb62449ba69b4a9c2e349b31 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240501 20:01:50.645256 31615 log.cc:924] Time spent T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.420s user 0.004s sys 0.000s
I20240501 20:01:50.631513 31553 consensus_queue.cc:260] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:50.646639 31553 raft_consensus.cc:3037] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:50.673931 31427 raft_consensus.cc:2726] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:01:50.674795 31640 raft_consensus.cc:482] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:50.678190 31421 raft_consensus.cc:482] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:50.678722 31421 raft_consensus.cc:504] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:50.620827 31434 raft_consensus.cc:504] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:50.685227 31427 raft_consensus.cc:482] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:50.685866 31427 raft_consensus.cc:504] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:50.686765 29657 raft_consensus.cc:2379] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d in current term 6: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:50.688803 31268 raft_consensus.cc:482] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:50.689373 31425 raft_consensus.cc:482] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:50.689266 31268 raft_consensus.cc:504] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:50.689836 31425 raft_consensus.cc:504] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:50.691406 31510 raft_consensus.cc:3037] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:01:50.689755 31153 raft_consensus.cc:482] T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:50.691999 31153 raft_consensus.cc:504] T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:50.700381 29734 raft_consensus.cc:1226] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 3. Current term is 4. Ops: []
W20240501 20:01:50.707260 29625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 80396.8us
I20240501 20:01:50.707952 29557 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33356 (request call id 54) took 3307 ms. Trace:
I20240501 20:01:50.708380 29625 leader_election.cc:302] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:50.675470 31640 raft_consensus.cc:504] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:50.726295 29655 raft_consensus.cc:1261] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240501 20:01:50.733455 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac2171683884f6f97cdfeb0a656cb49" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:50.734256 29805 raft_consensus.cc:2454] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 5.
I20240501 20:01:50.738189 29726 raft_consensus.cc:1226] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 3. Current term is 4. Ops: []
I20240501 20:01:50.712291 31434 leader_election.cc:288] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:50.744554 29807 raft_consensus.cc:1261] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240501 20:01:50.744833 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3f29f0eeea4b119984634202e5d9cd" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:50.718209 31472 leader_election.cc:288] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:50.746316 31449 leader_election.cc:288] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:50.708472 29557 rpcz_store.cc:276] 0501 20:01:47.400868 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:47.407311 (+ 6443us) service_pool.cc:225] Handling call
0501 20:01:50.707893 (+3300582us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0501 20:01:49.935537 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 20:01:49.936024 (+ 487us) write_op.cc:424] Acquiring schema lock in shared mode
0501 20:01:49.937526 (+ 1502us) write_op.cc:427] Acquired schema lock
0501 20:01:49.944174 (+ 6648us) tablet.cc:660] Decoding operations
0501 20:01:49.985126 (+ 40952us) write_op.cc:612] Acquiring the partition lock for write op
0501 20:01:49.985217 (+ 91us) write_op.cc:633] Partition lock acquired for write op
0501 20:01:49.985245 (+ 28us) tablet.cc:683] Acquiring locks for 1 operations
0501 20:01:49.985416 (+ 171us) tablet.cc:699] Row locks acquired
0501 20:01:49.985432 (+ 16us) write_op.cc:260] PREPARE: finished
0501 20:01:49.985554 (+ 122us) write_op.cc:270] Start()
0501 20:01:49.985651 (+ 97us) write_op.cc:276] Timestamp: P: 1714593709985516 usec, L: 0
0501 20:01:49.985673 (+ 22us) op_driver.cc:331] REPLICATION: starting
0501 20:01:49.986243 (+ 570us) log.cc:842] Serialized 538 byte log entry
0501 20:01:50.096173 (+109930us) op_driver.cc:444] REPLICATION: finished
0501 20:01:50.227471 (+131298us) write_op.cc:301] APPLY: starting
0501 20:01:50.227576 (+ 105us) tablet.cc:1365] starting BulkCheckPresence
0501 20:01:50.227679 (+ 103us) tablet.cc:1368] finished BulkCheckPresence
0501 20:01:50.228747 (+ 1068us) tablet.cc:1370] starting ApplyRowOperation cycle
0501 20:01:50.228962 (+ 215us) tablet.cc:1381] finished ApplyRowOperation cycle
0501 20:01:50.229135 (+ 173us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0501 20:01:50.229188 (+ 53us) write_op.cc:312] APPLY: finished
0501 20:01:50.229364 (+ 176us) log.cc:842] Serialized 22 byte log entry
0501 20:01:50.241492 (+ 12128us) write_op.cc:481] Releasing partition, row and schema locks
0501 20:01:50.248337 (+ 6845us) write_op.cc:446] Released schema lock
0501 20:01:50.249443 (+ 1106us) write_op.cc:341] FINISH: Updating metrics
0501 20:01:50.271039 (+ 21596us) write_op.cc:481] Releasing partition, row and schema locks
0501 20:01:50.273691 (+ 2652us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":1177286,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":131037,"apply.run_cpu_time_us":1930,"apply.run_wall_time_us":48775,"num_ops":1,"prepare.queue_time_us":269887,"prepare.run_cpu_time_us":10141,"prepare.run_wall_time_us":79247,"raft.queue_time_us":72214,"raft.run_cpu_time_us":19338,"raft.run_wall_time_us":99152,"replication_time_us":110407,"thread_start_us":290925,"threads_started":4,"wal-append.queue_time_us":20629}]]}
I20240501 20:01:50.746022 31425 leader_election.cc:288] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:50.746129 31153 leader_election.cc:288] T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:50.735914 31447 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20240501 20:01:50.747828 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649b71f9f267437ca51c758dd9ca6987" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:50.754480 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:50.753340 29807 raft_consensus.cc:1261] T 48630ae6eb394ac1b54955dd01cf0ed2 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20240501 20:01:50.760092 29625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 51786.7us
I20240501 20:01:50.761025 29625 leader_election.cc:302] T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:50.763465 29654 raft_consensus.cc:1261] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 7 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20240501 20:01:50.775753 29772 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 148715us
I20240501 20:01:50.776767 29772 leader_election.cc:302] T 456035a1cb3f4f7da52d0fc1dd1c656c P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:50.775456 31449 raft_consensus.cc:2726] T 49da4246f5f6438abe0e7e8c9a523d80 P 4c8da1d580614449a636b855bf11b95f [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 20:01:50.746022 31427 leader_election.cc:288] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:50.740455 31421 leader_election.cc:288] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:50.782183 29772 leader_election.cc:302] T 84fa28fb55f44b61ba1a88ed775a78ba P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:50.736680 31470 leader_election.cc:288] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:50.785571 29772 leader_election.cc:302] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:50.784205 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38cbeb0dc0c14e3d9182101eea885625" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:50.718003 31645 consensus_queue.cc:1046] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.038s
I20240501 20:01:50.791494 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1a08b455b94ae089396a15c837ddc2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:50.787866 31634 leader_election.cc:288] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:50.784679 31472 raft_consensus.cc:2726] T 456035a1cb3f4f7da52d0fc1dd1c656c P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:50.784931 31433 raft_consensus.cc:482] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:50.785291 31425 raft_consensus.cc:2726] T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240501 20:01:50.797642 31471 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.559s user 0.006s sys 0.016s
I20240501 20:01:50.797349 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6797d82dda2948d8a7445fcd33e44752" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:50.798803 29625 leader_election.cc:302] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:50.776484 31268 leader_election.cc:288] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:50.778863 31525 raft_consensus.cc:482] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:50.807590 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5525f546426c47a38edf7f56ad04b7ab" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:50.812517 29735 raft_consensus.cc:1261] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240501 20:01:50.814363 29700 leader_election.cc:302] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:50.809357 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668b4a4d782c4278932dd8413dc3ab3e" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:50.796197 31445 raft_consensus.cc:2726] T 84fa28fb55f44b61ba1a88ed775a78ba P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:01:50.799165 31470 raft_consensus.cc:2726] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240501 20:01:50.815734 29807 raft_consensus.cc:2454] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 5.
I20240501 20:01:50.794018 29772 leader_election.cc:302] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
W20240501 20:01:50.802876 31498 consensus_meta.cc:287] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d: Time spent flushing consensus metadata: real 0.530s user 0.000s sys 0.014s
I20240501 20:01:50.818738 29625 leader_election.cc:302] T 1ca3948261374cef8b1b206a43c00173 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:50.819985 29772 leader_election.cc:302] T 48630ae6eb394ac1b54955dd01cf0ed2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:50.821575 29772 leader_election.cc:302] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:50.824095 29698 leader_election.cc:302] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
W20240501 20:01:50.807936 31333 consensus_meta.cc:287] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f: Time spent flushing consensus metadata: real 0.536s user 0.015s sys 0.001s
I20240501 20:01:50.831382 29652 raft_consensus.cc:2396] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
W20240501 20:01:50.810557 31471 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.582s user 0.007s sys 0.016s
I20240501 20:01:50.807713 31153 raft_consensus.cc:482] T 2d86f8848a184a87947fd927c44e5a35 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:50.833117 29772 leader_election.cc:302] T 3c5345f6eb62449ba69b4a9c2e349b31 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:50.827446 31605 raft_consensus.cc:2781] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader pre-election won for term 6
W20240501 20:01:50.837699 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 218201us
I20240501 20:01:50.827508 31472 raft_consensus.cc:2726] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240501 20:01:50.807597 31426 consensus_queue.cc:1046] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20240501 20:01:50.827191 31427 raft_consensus.cc:482] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:50.806378 31433 raft_consensus.cc:504] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:50.828415 31425 raft_consensus.cc:2726] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240501 20:01:50.818390 31525 raft_consensus.cc:504] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:50.833657 31434 raft_consensus.cc:2726] T 1ca3948261374cef8b1b206a43c00173 P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:01:50.841451 29772 leader_election.cc:302] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:50.828291 31648 raft_consensus.cc:2726] T 48630ae6eb394ac1b54955dd01cf0ed2 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:01:50.836833 31153 raft_consensus.cc:504] T 2d86f8848a184a87947fd927c44e5a35 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:50.851637 31153 leader_election.cc:288] T 2d86f8848a184a87947fd927c44e5a35 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:50.838315 31445 raft_consensus.cc:2726] T 3c5345f6eb62449ba69b4a9c2e349b31 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:50.836921 31087 raft_consensus.cc:2726] T 8edcd903224d48f889b46e1b5318e0ec P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240501 20:01:50.833781 31470 raft_consensus.cc:2726] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:50.865435 31525 leader_election.cc:288] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:50.840186 31605 raft_consensus.cc:482] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:50.828161 31498 raft_consensus.cc:504] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:50.834556 31511 consensus_queue.cc:1035] T 6e38a2e91a3d48d09a22abce723af833 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
I20240501 20:01:50.834453 31043 consensus_queue.cc:1035] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.021s
I20240501 20:01:50.875586 31640 leader_election.cc:288] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:50.876706 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 44084.1us
I20240501 20:01:50.878095 29654 raft_consensus.cc:1261] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20240501 20:01:50.898682 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 74721.8us
I20240501 20:01:50.856094 31472 raft_consensus.cc:2726] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240501 20:01:50.906545 29659 raft_consensus.cc:1261] T bbcf30cf66714efdac997862dafbe4a2 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240501 20:01:50.904585 29698 leader_election.cc:302] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:50.908200 29658 raft_consensus.cc:3032] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:01:50.908429 29658 raft_consensus.cc:729] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:50.908856 29658 consensus_queue.cc:260] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:50.852653 31512 consensus_queue.cc:1035] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.050s
I20240501 20:01:50.901465 31043 consensus_queue.cc:1035] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.011s
I20240501 20:01:50.846252 31427 raft_consensus.cc:504] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:50.914019 31433 leader_election.cc:288] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:50.915340 31427 leader_election.cc:288] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:50.876425 31605 raft_consensus.cc:3037] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:50.917290 31634 raft_consensus.cc:482] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:50.917739 31634 raft_consensus.cc:504] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:50.919486 31634 leader_election.cc:288] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:50.923583 29698 leader_election.cc:302] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
W20240501 20:01:50.926582 29772 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 59581us
I20240501 20:01:50.926667 29700 leader_election.cc:302] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:50.927524 29658 raft_consensus.cc:3037] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:50.928421 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:01:50.870581 31135 consensus_queue.cc:1035] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.043s
I20240501 20:01:50.929647 31043 raft_consensus.cc:3037] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:50.939467 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "456035a1cb3f4f7da52d0fc1dd1c656c" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:50.931067 30414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e89c0572e4ee7be8e5315b459bdb3" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c" is_pre_election: true
I20240501 20:01:50.941116 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b337b8e3e38d4ad6b1076fbd0d0a1609" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:50.941169 29804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5525fc4b82a44b0991c3dd6098fde81" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:50.942229 29803 raft_consensus.cc:1261] T bbcf30cf66714efdac997862dafbe4a2 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240501 20:01:50.942516 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03f5d1686214485d91a0baf4b870740e" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:50.944118 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb27193abbf4644a48662909bc7b04a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:50.945307 31307 raft_consensus.cc:482] T f6d88139f89c47ff84cad674147aae0f P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:50.945775 31307 raft_consensus.cc:504] T f6d88139f89c47ff84cad674147aae0f P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:50.947350 31307 leader_election.cc:288] T f6d88139f89c47ff84cad674147aae0f P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:50.947731 31307 raft_consensus.cc:482] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:50.948019 31307 raft_consensus.cc:504] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:50.949278 31498 leader_election.cc:288] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:50.931259 29698 leader_election.cc:302] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:50.931282 31659 raft_consensus.cc:2781] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240501 20:01:50.932201 31488 raft_consensus.cc:2781] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240501 20:01:50.872092 31470 consensus_queue.cc:1035] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.050s
I20240501 20:01:50.951083 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfac893b4b4f457694a4eb184b6abb53" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:50.951745 29698 leader_election.cc:302] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:50.878850 31648 consensus_queue.cc:1035] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.048s
I20240501 20:01:50.877321 31063 raft_consensus.cc:504] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:50.949971 31307 leader_election.cc:288] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:50.973691 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af08384057a4d85977481472f460168" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:50.973845 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "add9d7579351483fb268f89a44e95d8a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:50.976507 29698 leader_election.cc:302] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:50.978745 29625 leader_election.cc:302] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:50.979369 29658 raft_consensus.cc:1261] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240501 20:01:50.980563 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d86f8848a184a87947fd927c44e5a35" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:50.981524 29658 raft_consensus.cc:2396] T 2d86f8848a184a87947fd927c44e5a35 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20240501 20:01:50.956431 31488 raft_consensus.cc:482] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:50.982722 31488 raft_consensus.cc:3037] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:50.984300 31043 raft_consensus.cc:2726] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240501 20:01:50.984903 29802 raft_consensus.cc:1226] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 4. Current term is 5. Ops: []
I20240501 20:01:50.985965 31063 leader_election.cc:288] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:50.986569 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:50.988780 29653 raft_consensus.cc:1261] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240501 20:01:50.987993 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c597f15e844f30837012baed4b0501" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:50.990204 29800 raft_consensus.cc:1226] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 5. Current term is 6. Ops: []
I20240501 20:01:50.962604 31658 raft_consensus.cc:2726] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240501 20:01:50.989873 29652 raft_consensus.cc:2362] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 6. Re-sending same reply.
I20240501 20:01:50.921826 31639 raft_consensus.cc:504] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:51.000942 29654 raft_consensus.cc:3032] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [term 5 LEADER]: Stepping down as leader of term 5
I20240501 20:01:51.001296 29654 raft_consensus.cc:729] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:51.001538 31639 leader_election.cc:288] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:51.003124 29801 raft_consensus.cc:1226] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 4. Current term is 5. Ops: []
I20240501 20:01:51.005560 29656 raft_consensus.cc:1226] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 5. Current term is 6. Ops: []
I20240501 20:01:51.001981 29654 consensus_queue.cc:260] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } }
I20240501 20:01:51.007601 29654 raft_consensus.cc:3037] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:51.009320 31087 raft_consensus.cc:2781] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader election won for term 4
W20240501 20:01:51.009415 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 154175us
I20240501 20:01:51.013269 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d86f8848a184a87947fd927c44e5a35" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:51.013468 29728 raft_consensus.cc:1203] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [term 6 LEADER]: Deduplicated request from leader. Original: 4.3->[5.4-5.4] Dedup: 5.4->[]
I20240501 20:01:51.013911 29728 raft_consensus.cc:1226] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [term 6 LEADER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20240501 20:01:51.009747 29698 leader_election.cc:302] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:50.956255 31659 raft_consensus.cc:482] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting leader election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:50.958010 31640 raft_consensus.cc:2726] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240501 20:01:51.027598 29698 leader_election.cc:302] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:51.042147 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668b4a4d782c4278932dd8413dc3ab3e" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:51.041638 29698 leader_election.cc:302] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:51.045794 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649b71f9f267437ca51c758dd9ca6987" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:50.968541 31525 consensus_queue.cc:1035] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.073s
I20240501 20:01:51.049013 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a3f29f0eeea4b119984634202e5d9cd" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:51.049396 29802 raft_consensus.cc:1261] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240501 20:01:51.041253 31659 raft_consensus.cc:3037] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:51.052541 29802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6797d82dda2948d8a7445fcd33e44752" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:51.040690 31606 raft_consensus.cc:2726] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240501 20:01:51.053913 31043 raft_consensus.cc:2726] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240501 20:01:51.041615 31658 raft_consensus.cc:2781] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240501 20:01:51.056182 31658 raft_consensus.cc:482] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting leader election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:51.056521 31658 raft_consensus.cc:3037] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:51.057912 29802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38cbeb0dc0c14e3d9182101eea885625" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:51.059217 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:51.043011 31628 consensus_queue.cc:1035] T a8e9f75387564275a60d38e9a910319c P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
I20240501 20:01:51.045571 31087 raft_consensus.cc:686] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:51.060439 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec073fe645614fe28e7837de3c22510b" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:51.061627 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c1a08b455b94ae089396a15c837ddc2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:51.061378 31087 consensus_queue.cc:237] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:51.061990 31488 raft_consensus.cc:504] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:51.062696 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d86f8848a184a87947fd927c44e5a35" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:51.066042 29698 leader_election.cc:302] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:51.066165 31506 consensus_queue.cc:260] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:51.066439 31605 raft_consensus.cc:504] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:51.067940 29651 raft_consensus.cc:2454] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 7.
I20240501 20:01:51.063858 31488 leader_election.cc:288] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:51.133605 29806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5525f546426c47a38edf7f56ad04b7ab" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:51.134691 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a35893b91247a99912eba787d787cf" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
W20240501 20:01:51.136013 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 45351.5us
I20240501 20:01:51.092386 31628 raft_consensus.cc:2781] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Leader election won for term 6
I20240501 20:01:51.112735 31506 raft_consensus.cc:3037] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:51.138105 29654 raft_consensus.cc:1261] T 69915dd5992641e3a24f7f45f1cd5c0c P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240501 20:01:51.112438 31043 consensus_queue.cc:1035] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20240501 20:01:51.112797 31525 consensus_queue.cc:1035] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.050s
I20240501 20:01:51.152170 31135 raft_consensus.cc:482] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:51.156970 31639 raft_consensus.cc:482] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240501 20:01:51.153708 30383 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 110540us
W20240501 20:01:51.186084 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 143673us
I20240501 20:01:51.205415 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668b4a4d782c4278932dd8413dc3ab3e" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:51.206375 29805 raft_consensus.cc:3037] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:51.223591 29660 raft_consensus.cc:1261] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 7 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20240501 20:01:51.181324 31664 log.cc:924] Time spent T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d: Append to log took a long time: real 0.220s user 0.000s sys 0.001s
W20240501 20:01:51.235091 29772 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 81324.8us
I20240501 20:01:51.172116 31135 raft_consensus.cc:504] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:51.241156 29803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac2171683884f6f97cdfeb0a656cb49" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
W20240501 20:01:51.221421 30385 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 177686us
I20240501 20:01:51.246487 31628 raft_consensus.cc:686] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 6 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:51.252501 31628 consensus_queue.cc:237] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:51.254315 29805 raft_consensus.cc:2454] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 6.
I20240501 20:01:51.184218 31639 raft_consensus.cc:504] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:51.261123 29700 leader_election.cc:302] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:51.230731 31640 consensus_queue.cc:1035] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.020s
I20240501 20:01:51.242326 31605 leader_election.cc:288] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:51.267489 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 124891us
I20240501 20:01:51.289527 29802 raft_consensus.cc:1203] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Deduplicated request from leader. Original: 4.3->[5.4-5.4] Dedup: 5.4->[]
I20240501 20:01:51.290609 29802 raft_consensus.cc:1226] T 69915dd5992641e3a24f7f45f1cd5c0c P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 5. Current term is 6. Ops: [5.4-5.4]
W20240501 20:01:51.292519 31517 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20240501 20:01:51.293277 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac2171683884f6f97cdfeb0a656cb49" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
W20240501 20:01:51.288720 31666 log.cc:924] Time spent T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d: Append to log took a long time: real 0.056s user 0.001s sys 0.000s
I20240501 20:01:51.295775 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48fd0a8e289401aa7417c42ff2c81b2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:51.296399 29731 raft_consensus.cc:2454] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 6.
I20240501 20:01:51.297999 31467 leader_election.cc:288] T 010ceabf3e5a4b1c9330b80dade765bb P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:51.299041 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e38a2e91a3d48d09a22abce723af833" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:51.304841 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77be7101c0fe4085b6706296f609870a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:51.304688 31135 leader_election.cc:288] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:51.306253 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e49dd5180f04313ab33942fb3ddc147" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:51.301048 31605 raft_consensus.cc:2781] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Leader election won for term 6
I20240501 20:01:51.360757 29554 catalog_manager.cc:5484] T 03f5d1686214485d91a0baf4b870740e P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:51.365865 29730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f261d2a05bf4c838532049601694b4a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:51.327205 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d88139f89c47ff84cad674147aae0f" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:51.327930 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9472c53f79ec423f82c396bd980674e2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:51.366802 29726 raft_consensus.cc:2454] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 4.
I20240501 20:01:51.364686 29554 catalog_manager.cc:5484] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:51.370857 29554 catalog_manager.cc:5484] T 3c5345f6eb62449ba69b4a9c2e349b31 P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 2 to 3, leader changed from 4c8da1d580614449a636b855bf11b95f (127.27.217.193) to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 3 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:51.282675 31659 raft_consensus.cc:504] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:51.375129 29554 catalog_manager.cc:5484] T 458740ab5c444b6ab3d1991c69db5084 P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 3 to 4, leader changed from 4c8da1d580614449a636b855bf11b95f (127.27.217.193) to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 4 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:51.378228 29554 catalog_manager.cc:5484] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
W20240501 20:01:51.373337 31652 log.cc:924] Time spent T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.531s user 0.004s sys 0.001s
I20240501 20:01:51.380501 29554 catalog_manager.cc:5484] T 0fb27193abbf4644a48662909bc7b04a P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:51.383539 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010ceabf3e5a4b1c9330b80dade765bb" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:51.387132 29729 raft_consensus.cc:1226] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 5. Current term is 6. Ops: []
I20240501 20:01:51.389827 29554 catalog_manager.cc:5484] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } }
I20240501 20:01:51.398581 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec073fe645614fe28e7837de3c22510b" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:51.372917 31671 consensus_queue.cc:1046] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.019s
I20240501 20:01:51.400776 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9472c53f79ec423f82c396bd980674e2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:51.401451 29727 raft_consensus.cc:2379] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 4: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:51.382234 31605 raft_consensus.cc:686] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [term 6 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:51.403236 29731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07bdb8b09b8c4ef4b8e49daa9b8c01d6" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:51.404821 29731 raft_consensus.cc:2454] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 6.
I20240501 20:01:51.406438 31500 consensus_queue.cc:1046] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:51.407579 31500 consensus_queue.cc:1035] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:51.408177 31500 consensus_queue.cc:1225] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Peer 4c8da1d580614449a636b855bf11b95f log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20240501 20:01:51.409090 31500 raft_consensus.cc:3032] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [term 5 LEADER]: Stepping down as leader of term 5
I20240501 20:01:51.409440 31500 raft_consensus.cc:729] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:51.410113 31500 consensus_queue.cc:260] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:51.410686 29625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 431955us
I20240501 20:01:51.411092 31500 raft_consensus.cc:3037] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:51.412161 31063 raft_consensus.cc:2726] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240501 20:01:51.412377 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a35893b91247a99912eba787d787cf" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
W20240501 20:01:51.415410 31649 log.cc:924] Time spent T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 0.577s user 0.009s sys 0.000s
I20240501 20:01:51.417044 31658 raft_consensus.cc:504] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:51.418380 29735 raft_consensus.cc:1226] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 5. Current term is 6. Ops: []
I20240501 20:01:51.419301 31087 raft_consensus.cc:3032] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Stepping down as leader of term 4
I20240501 20:01:51.419728 31087 raft_consensus.cc:729] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
W20240501 20:01:51.420804 31662 log.cc:924] Time spent T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d: Append to log took a long time: real 0.182s user 0.002s sys 0.003s
I20240501 20:01:51.420538 31087 consensus_queue.cc:260] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:51.421736 31087 raft_consensus.cc:3037] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:51.437829 31639 leader_election.cc:288] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:51.403506 29554 catalog_manager.cc:5484] T 1ca3948261374cef8b1b206a43c00173 P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 1 to 2, leader changed from 4c8da1d580614449a636b855bf11b95f (127.27.217.193) to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 2 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:51.439471 29734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36270 (request call id 1096) took 1157 ms. Trace:
I20240501 20:01:51.439118 29554 catalog_manager.cc:5484] T 49da4246f5f6438abe0e7e8c9a523d80 P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:51.439925 29734 rpcz_store.cc:276] 0501 20:01:50.281897 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:50.282030 (+ 133us) service_pool.cc:225] Handling call
0501 20:01:50.282299 (+ 269us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:50.700195 (+417896us) spinlock_profiling.cc:243] Waited 11.2 ms on lock 0x7b5400279fa0. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe457800d30 00007fe45780b973 00007fe45787d550 00007fe45787e5c1 00007fe45787b12f 00007fe45787abee 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc
0501 20:01:50.700893 (+ 698us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:51.439181 (+738288us) spinlock_profiling.cc:243] Waited 39.9 ms on lock 0x7b54003714d4. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787c1a1 00007fe45787abee 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc 00007fe455e9e782 00007fe455ca2bcc 00007fe455ca2006
0501 20:01:51.439403 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":117553920}
I20240501 20:01:51.447218 29625 leader_election.cc:302] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:51.453023 29651 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "24a2bba62f434d1c8489b55c8fec2187"
dest_uuid: "4c8da1d580614449a636b855bf11b95f"
mode: GRACEFUL
new_leader_uuid: "ee0d97247d394a8facfa774f15f3a67d"
from {username='slave'} at 127.0.0.1:60736
I20240501 20:01:51.453639 29651 raft_consensus.cc:595] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Received request to transfer leadership to TS ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:51.454030 29651 raft_consensus.cc:603] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240501 20:01:51.455822 31413 raft_consensus.cc:1050] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f: attempting to promote NON_VOTER 888d99522e4844118f8f1cf26f268c8c to VOTER
I20240501 20:01:51.455960 31605 consensus_queue.cc:237] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:51.456460 29773 leader_election.cc:302] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
W20240501 20:01:51.457999 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 55890.8us
I20240501 20:01:51.458223 31135 raft_consensus.cc:2781] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240501 20:01:51.458690 31135 raft_consensus.cc:482] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:51.459033 31135 raft_consensus.cc:3037] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:01:51.461742 31659 leader_election.cc:288] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:51.465008 29654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e38a2e91a3d48d09a22abce723af833" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:51.466616 29654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7e89c0572e4ee7be8e5315b459bdb3" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:51.419337 31658 leader_election.cc:288] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:51.470304 29653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac2171683884f6f97cdfeb0a656cb49" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:51.473834 29653 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc"
dest_uuid: "4c8da1d580614449a636b855bf11b95f"
from {username='slave'} at 127.0.0.1:59136
I20240501 20:01:51.474298 29653 raft_consensus.cc:482] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:01:51.474648 29653 raft_consensus.cc:3037] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:01:51.474289 31135 raft_consensus.cc:504] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:51.476481 29654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48fd0a8e289401aa7417c42ff2c81b2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:51.477110 29660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59136 (request call id 933) took 1132 ms. Trace:
I20240501 20:01:51.477808 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9472c53f79ec423f82c396bd980674e2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:51.478410 29656 raft_consensus.cc:2454] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 4.
I20240501 20:01:51.481065 29625 leader_election.cc:302] T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:51.481422 29773 leader_election.cc:302] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:51.481848 31135 leader_election.cc:288] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 7 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:51.481899 31433 raft_consensus.cc:2726] T 03f5d1686214485d91a0baf4b870740e P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240501 20:01:51.483275 29625 leader_election.cc:302] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:51.483971 31433 raft_consensus.cc:2726] T 0fb27193abbf4644a48662909bc7b04a P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240501 20:01:51.485473 29654 raft_consensus.cc:2454] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 6.
I20240501 20:01:51.485026 29625 leader_election.cc:302] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:51.486477 31506 raft_consensus.cc:2726] T bfac893b4b4f457694a4eb184b6abb53 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240501 20:01:51.482264 31501 raft_consensus.cc:2781] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240501 20:01:51.493499 31501 raft_consensus.cc:482] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:51.493870 31501 raft_consensus.cc:3037] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:51.495651 31413 consensus_queue.cc:237] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 4, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:51.481431 31510 raft_consensus.cc:2726] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240501 20:01:51.477465 29660 rpcz_store.cc:276] 0501 20:01:50.344794 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:50.449481 (+104687us) service_pool.cc:225] Handling call
0501 20:01:51.223282 (+773801us) raft_consensus.cc:1457] Updating replica for 0 ops
0501 20:01:51.224016 (+ 734us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:51.476934 (+252918us) spinlock_profiling.cc:243] Waited 773 ms on lock 0x7b54003953d0. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe4578006b1 00007fe45787ac59 00007fe45c7bd099 00007fe455e9f694 00007fe455e9f622 00007fe455e9f551 00007fe455e9f4cc 00007fe455e9e782 00007fe455ca2bcc 00007fe455ca2006 00007fe455ca19af
0501 20:01:51.477054 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1778804096}
I20240501 20:01:51.503024 29773 leader_election.cc:302] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:51.503981 31641 raft_consensus.cc:2781] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240501 20:01:51.504467 31641 raft_consensus.cc:482] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:51.504858 31641 raft_consensus.cc:3037] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:01:51.506076 29654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77be7101c0fe4085b6706296f609870a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:51.508037 29655 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37936 (request call id 134) took 1120 ms. Trace:
I20240501 20:01:51.509115 31135 consensus_queue.cc:1046] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240501 20:01:51.509614 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e49dd5180f04313ab33942fb3ddc147" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:51.508423 29655 rpcz_store.cc:276] 0501 20:01:50.387411 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:50.730756 (+343345us) service_pool.cc:225] Handling call
0501 20:01:50.731085 (+ 329us) raft_consensus.cc:1457] Updating replica for 1 ops
0501 20:01:50.735524 (+ 4439us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:50.735564 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0501 20:01:50.742933 (+ 7369us) log.cc:842] Serialized 26 byte log entry
0501 20:01:50.834068 (+ 91135us) raft_consensus.cc:1631] Marking committed up to 8
0501 20:01:50.848498 (+ 14430us) log.cc:842] Serialized 14 byte log entry
0501 20:01:51.386399 (+537901us) spinlock_profiling.cc:243] Waited 1.57 ms on lock 0x7b18000da920. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe45d7f2661 00007fe45d7ff3e1 00007fe4548ca2ea 00007fe45dfb9fe5 00007fe45494e3d4 00007fe45494dcc5 00007fe457873620 00007fe4578857c6 00007fe457896327 00007fe4578962a8 00007fe457896228
0501 20:01:51.386554 (+ 155us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:51.386621 (+ 67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0501 20:01:51.471633 (+ 85012us) spinlock_profiling.cc:243] Waited 11.8 ms on lock 0x7b18000da920. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe45d7f2661 00007fe45d7ff3e1 00007fe4549511b8 00007fe454954691 00007fe454954649 00007fe4549545d9 00007fe4549545a1 00007fe45495389d 00007fe45dbc5da4 00007fe45dbc5bd9 00007fe454933936
0501 20:01:51.506098 (+ 34465us) spinlock_profiling.cc:243] Waited 33.8 ms on lock 0x7b44000232cc. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe454797b51 00007fe454865685 00007fe4548660c2 00007fe45750e441 00007fe457509d4a 00007fe4574fdb1e 00007fe4574fce66 00007fe4574fca88 00007fe457506481 00007fe457506439 00007fe4575063c9
0501 20:01:51.507854 (+ 1756us) raft_consensus.cc:1671] finished
0501 20:01:51.507871 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:51.507981 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":29330,"prepare.queue_time_us":21206,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":29,"raft.queue_time_us":544530,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":13002,"spinlock_wait_cycles":108424064,"thread_start_us":49307,"threads_started":1,"wal-append.queue_time_us":78972}
I20240501 20:01:51.510185 29658 raft_consensus.cc:2454] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 6.
I20240501 20:01:51.510718 29735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211e27a3b0084b89a6dd7550849b5a65" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:51.511307 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48fd0a8e289401aa7417c42ff2c81b2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:51.511835 29734 raft_consensus.cc:3037] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:01:51.515641 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d88139f89c47ff84cad674147aae0f" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:51.517863 29734 raft_consensus.cc:2454] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 7.
I20240501 20:01:51.518302 29654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f261d2a05bf4c838532049601694b4a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
W20240501 20:01:51.518695 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 117821us
I20240501 20:01:51.509068 29625 leader_election.cc:302] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:51.520124 29625 leader_election.cc:302] T add9d7579351483fb268f89a44e95d8a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: 888d99522e4844118f8f1cf26f268c8c, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:51.491752 29653 raft_consensus.cc:504] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:51.521761 29656 raft_consensus.cc:1226] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [term 7 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 5. Current term is 7. Ops: []
I20240501 20:01:51.526860 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5525fc4b82a44b0991c3dd6098fde81" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:51.528115 29653 leader_election.cc:288] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 7 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:51.532675 29729 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3af08384057a4d85977481472f460168"
dest_uuid: "fdfeac4190e045f290e003b2298d988d"
mode: GRACEFUL
new_leader_uuid: "ee0d97247d394a8facfa774f15f3a67d"
from {username='slave'} at 127.0.0.1:46074
I20240501 20:01:51.533365 29729 raft_consensus.cc:595] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [term 3 LEADER]: Received request to transfer leadership to TS ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:51.540678 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece8d4ed84c34b748f7b6272b31c90ac" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:51.568105 29625 leader_election.cc:302] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:51.559211 29773 leader_election.cc:302] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:51.569672 29625 leader_election.cc:302] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:51.573555 29809 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5525f546426c47a38edf7f56ad04b7ab"
dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
mode: GRACEFUL
new_leader_uuid: "4c8da1d580614449a636b855bf11b95f"
from {username='slave'} at 127.0.0.1:33582
I20240501 20:01:51.574415 29809 raft_consensus.cc:595] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [term 5 LEADER]: Received request to transfer leadership to TS 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:51.524993 31510 raft_consensus.cc:2726] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:51.525346 31506 raft_consensus.cc:2726] T add9d7579351483fb268f89a44e95d8a P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:01:51.587998 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010ceabf3e5a4b1c9330b80dade765bb" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:51.583191 29625 leader_election.cc:302] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:51.590544 29625 leader_election.cc:302] T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:51.558723 31498 raft_consensus.cc:3032] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [term 5 LEADER]: Stepping down as leader of term 5
I20240501 20:01:51.591696 31647 raft_consensus.cc:2781] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 7 FOLLOWER]: Leader election won for term 7
I20240501 20:01:51.592480 31498 raft_consensus.cc:729] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:51.592684 31647 raft_consensus.cc:686] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [term 7 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:51.593310 31498 consensus_queue.cc:260] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:51.593493 31647 consensus_queue.cc:237] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:51.594535 31498 raft_consensus.cc:3037] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:51.524824 31433 consensus_queue.cc:1046] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.053s
I20240501 20:01:51.580826 31268 raft_consensus.cc:2726] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240501 20:01:51.614789 29660 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a8e9f75387564275a60d38e9a910319c"
dest_uuid: "4c8da1d580614449a636b855bf11b95f"
mode: GRACEFUL
new_leader_uuid: "ee0d97247d394a8facfa774f15f3a67d"
from {username='slave'} at 127.0.0.1:60736
I20240501 20:01:51.615629 29660 raft_consensus.cc:595] T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Received request to transfer leadership to TS ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:51.616029 29660 raft_consensus.cc:603] T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240501 20:01:51.618391 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07bdb8b09b8c4ef4b8e49daa9b8c01d6" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:51.556381 31634 consensus_queue.cc:1046] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.040s
I20240501 20:01:51.620833 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:51.625389 29660 raft_consensus.cc:2362] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 7. Re-sending same reply.
I20240501 20:01:51.622509 29625 leader_election.cc:302] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:51.626235 29654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211e27a3b0084b89a6dd7550849b5a65" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:51.627169 29654 raft_consensus.cc:2379] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d in current term 4: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:51.586622 31448 raft_consensus.cc:2726] T 7a3f29f0eeea4b119984634202e5d9cd P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240501 20:01:51.657021 29654 pending_rounds.cc:76] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f: Aborting all ops after (but not including) 2
I20240501 20:01:51.657480 29654 pending_rounds.cc:97] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20240501 20:01:51.657788 29654 raft_consensus.cc:2866] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 20:01:51.615404 31413 raft_consensus.cc:2726] T 6797d82dda2948d8a7445fcd33e44752 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240501 20:01:51.661391 29625 leader_election.cc:302] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:51.671711 29625 leader_election.cc:302] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:51.673135 29625 leader_election.cc:302] T 2d86f8848a184a87947fd927c44e5a35 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:51.675814 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48fd0a8e289401aa7417c42ff2c81b2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:51.584409 31421 consensus_queue.cc:1035] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.071s
I20240501 20:01:51.676481 29656 raft_consensus.cc:3037] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:01:51.635236 31471 raft_consensus.cc:2726] T 38cbeb0dc0c14e3d9182101eea885625 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240501 20:01:51.683302 29729 raft_consensus.cc:1226] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 5. Current term is 6. Ops: []
I20240501 20:01:51.684224 29772 leader_election.cc:302] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:51.685691 29772 leader_election.cc:302] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, 888d99522e4844118f8f1cf26f268c8c
I20240501 20:01:51.688130 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:51.688833 29727 raft_consensus.cc:3037] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Advancing to term 7
I20240501 20:01:51.655731 30417 raft_consensus.cc:1261] T be383be783234855878cb8fd442c7b05 P 888d99522e4844118f8f1cf26f268c8c [term 4 LEARNER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240501 20:01:51.692209 29660 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aa8cd40cb4044d18a86001ce6ddf97ab"
dest_uuid: "4c8da1d580614449a636b855bf11b95f"
mode: GRACEFUL
new_leader_uuid: "fdfeac4190e045f290e003b2298d988d"
from {username='slave'} at 127.0.0.1:60736
I20240501 20:01:51.695866 29660 raft_consensus.cc:595] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Received request to transfer leadership to TS fdfeac4190e045f290e003b2298d988d
I20240501 20:01:51.696363 29660 raft_consensus.cc:603] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240501 20:01:51.669322 31063 raft_consensus.cc:2726] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240501 20:01:51.692879 29772 leader_election.cc:302] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:51.694749 29730 raft_consensus.cc:1226] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [term 6 LEADER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20240501 20:01:51.701974 29807 raft_consensus.cc:3037] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:51.703554 29625 leader_election.cc:302] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:51.692647 29728 raft_consensus.cc:1261] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240501 20:01:51.707330 29625 leader_election.cc:302] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:51.677193 31493 raft_consensus.cc:2726] T ec073fe645614fe28e7837de3c22510b P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:51.680294 31517 raft_consensus.cc:2726] T 3c1a08b455b94ae089396a15c837ddc2 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240501 20:01:51.680131 31510 raft_consensus.cc:2726] T 2d86f8848a184a87947fd927c44e5a35 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240501 20:01:51.660872 31641 raft_consensus.cc:504] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:51.715101 29727 raft_consensus.cc:2454] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 7.
I20240501 20:01:51.714848 29656 raft_consensus.cc:2454] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 7.
I20240501 20:01:51.720556 29772 leader_election.cc:302] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:51.722273 28519 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240501 20:01:51.726526 29807 raft_consensus.cc:1261] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240501 20:01:51.726552 28519 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:01:51.711088 31433 raft_consensus.cc:2726] T 77a35893b91247a99912eba787d787cf P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240501 20:01:51.698840 31307 raft_consensus.cc:2726] T 6e38a2e91a3d48d09a22abce723af833 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240501 20:01:51.727486 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:01:51.729816 29772 leader_election.cc:302] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters: fdfeac4190e045f290e003b2298d988d
I20240501 20:01:51.703079 31634 raft_consensus.cc:2726] T fa7e89c0572e4ee7be8e5315b459bdb3 P ee0d97247d394a8facfa774f15f3a67d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240501 20:01:51.729847 29698 leader_election.cc:302] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:51.708819 31470 raft_consensus.cc:2726] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240501 20:01:51.740645 29772 leader_election.cc:302] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:51.716408 31063 raft_consensus.cc:2726] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240501 20:01:51.752488 29655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36057b3f24d44b9aac3a1378053a395f" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:51.753232 31634 raft_consensus.cc:2726] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240501 20:01:51.753376 29655 raft_consensus.cc:2454] T 36057b3f24d44b9aac3a1378053a395f P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 4.
I20240501 20:01:51.760461 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "456035a1cb3f4f7da52d0fc1dd1c656c" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:51.762040 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece8d4ed84c34b748f7b6272b31c90ac" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:51.763332 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649b71f9f267437ca51c758dd9ca6987" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:51.752728 29772 leader_election.cc:302] T f6d88139f89c47ff84cad674147aae0f P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:51.768796 31470 raft_consensus.cc:2726] T f6d88139f89c47ff84cad674147aae0f P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:51.771339 31470 raft_consensus.cc:979] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d: : Instructing follower 4c8da1d580614449a636b855bf11b95f to start an election
I20240501 20:01:51.771868 31470 raft_consensus.cc:1067] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [term 5 LEADER]: Signalling peer 4c8da1d580614449a636b855bf11b95f to start an election
I20240501 20:01:51.772042 29652 raft_consensus.cc:2379] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d in current term 6: Already voted for candidate 4c8da1d580614449a636b855bf11b95f in this term.
I20240501 20:01:51.773063 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5525fc4b82a44b0991c3dd6098fde81" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:51.773172 31197 consensus_queue.cc:1046] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 20:01:51.774529 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668b4a4d782c4278932dd8413dc3ab3e" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:51.775823 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac2171683884f6f97cdfeb0a656cb49" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:51.776775 29656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d88139f89c47ff84cad674147aae0f" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:51.777281 29772 leader_election.cc:302] T 010ceabf3e5a4b1c9330b80dade765bb P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:51.728235 31449 raft_consensus.cc:482] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:51.731663 31639 raft_consensus.cc:2726] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20240501 20:01:51.787248 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 57280.8us
I20240501 20:01:51.788172 29624 leader_election.cc:302] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:51.744300 31307 raft_consensus.cc:2781] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240501 20:01:51.794304 29772 leader_election.cc:302] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:51.774969 31470 raft_consensus.cc:3032] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 5 LEADER]: Stepping down as leader of term 5
I20240501 20:01:51.734119 31531 consensus_queue.cc:1035] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.051s
I20240501 20:01:51.721344 31705 consensus_queue.cc:1035] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.029s
I20240501 20:01:51.734513 31447 consensus_queue.cc:1035] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.061s
I20240501 20:01:51.735430 31460 consensus_queue.cc:1035] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.058s
I20240501 20:01:51.781729 31472 raft_consensus.cc:2726] T 010ceabf3e5a4b1c9330b80dade765bb P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:01:51.798935 31470 raft_consensus.cc:729] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:51.798918 31648 raft_consensus.cc:2726] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:51.795943 31307 raft_consensus.cc:482] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:51.780933 31506 consensus_queue.cc:1046] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.025s
I20240501 20:01:51.800627 31433 raft_consensus.cc:2781] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [term 7 FOLLOWER]: Leader election won for term 7
I20240501 20:01:51.785136 31449 raft_consensus.cc:504] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:51.817999 31693 log.cc:924] Time spent T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.107s user 0.000s sys 0.000s
W20240501 20:01:51.819700 31694 log.cc:924] Time spent T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
I20240501 20:01:51.820638 31470 consensus_queue.cc:260] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:51.821897 31470 raft_consensus.cc:3037] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Advancing to term 7
I20240501 20:01:51.822248 31511 consensus_queue.cc:1035] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240501 20:01:51.820521 31449 leader_election.cc:288] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:51.838583 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9472c53f79ec423f82c396bd980674e2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:51.842479 29801 raft_consensus.cc:1226] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [term 7 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 6. Current term is 7. Ops: []
I20240501 20:01:51.842687 29808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:51.815826 31307 raft_consensus.cc:3037] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:01:51.851632 31135 raft_consensus.cc:3032] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [term 5 LEADER]: Stepping down as leader of term 5
I20240501 20:01:51.852273 31135 raft_consensus.cc:729] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:51.853312 31135 consensus_queue.cc:260] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:51.854846 31135 raft_consensus.cc:3037] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:51.857923 29801 raft_consensus.cc:1261] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240501 20:01:51.858042 29802 raft_consensus.cc:1226] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20240501 20:01:51.861444 31690 raft_consensus.cc:979] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d: : Instructing follower ee0d97247d394a8facfa774f15f3a67d to start an election
I20240501 20:01:51.864508 31501 raft_consensus.cc:504] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:51.866583 31501 leader_election.cc:288] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:51.874933 29807 raft_consensus.cc:2354] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f for earlier term 4. Current term is 5.
I20240501 20:01:51.877988 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f261d2a05bf4c838532049601694b4a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:51.878718 29807 raft_consensus.cc:2379] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 5: Already voted for candidate ee0d97247d394a8facfa774f15f3a67d in this term.
I20240501 20:01:51.880890 29809 raft_consensus.cc:1226] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Rejecting Update request from peer 4c8da1d580614449a636b855bf11b95f for earlier term 4. Current term is 5. Ops: []
I20240501 20:01:51.904870 29659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37936 (request call id 135) took 1412 ms. Trace:
I20240501 20:01:51.905467 29659 rpcz_store.cc:276] 0501 20:01:50.492973 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:50.912450 (+419477us) service_pool.cc:225] Handling call
0501 20:01:50.954067 (+ 41617us) raft_consensus.cc:1457] Updating replica for 1 ops
0501 20:01:51.409613 (+455546us) raft_consensus.cc:1520] Early marking committed up to index 0
0501 20:01:51.409643 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0501 20:01:51.410044 (+ 401us) log.cc:842] Serialized 26 byte log entry
0501 20:01:51.577735 (+167691us) raft_consensus.cc:1631] Marking committed up to 6
0501 20:01:51.797035 (+219300us) log.cc:842] Serialized 14 byte log entry
0501 20:01:51.904450 (+107415us) raft_consensus.cc:1683] Filling consensus response to leader.
0501 20:01:51.904562 (+ 112us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0501 20:01:51.904621 (+ 59us) raft_consensus.cc:1671] finished
0501 20:01:51.904640 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0501 20:01:51.904797 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":566578,"prepare.queue_time_us":208476,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":32,"raft.queue_time_us":564692,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":180,"thread_start_us":355265,"threads_started":2,"wal-append.queue_time_us":254540}
I20240501 20:01:51.911127 31153 raft_consensus.cc:3032] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [term 5 LEADER]: Stepping down as leader of term 5
I20240501 20:01:51.911752 31153 raft_consensus.cc:729] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:51.921299 31153 consensus_queue.cc:260] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:51.923013 31153 raft_consensus.cc:3037] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:51.914755 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07bdb8b09b8c4ef4b8e49daa9b8c01d6" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
W20240501 20:01:51.924607 31700 log.cc:924] Time spent T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d: Append to log took a long time: real 0.110s user 0.000s sys 0.001s
I20240501 20:01:51.884824 31641 leader_election.cc:288] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 7 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:51.907850 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9472c53f79ec423f82c396bd980674e2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:51.918747 29625 leader_election.cc:398] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 election: Vote denied by peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861) with higher term. Message: Invalid argument: T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f for earlier term 4. Current term is 5.
I20240501 20:01:51.931020 29625 leader_election.cc:401] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20240501 20:01:51.918812 31511 consensus_queue.cc:1046] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 20:01:51.919157 31448 consensus_queue.cc:1035] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:01:51.919821 31425 consensus_queue.cc:1046] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 20:01:51.930552 29659 raft_consensus.cc:3037] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:51.933322 29652 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5525f546426c47a38edf7f56ad04b7ab"
dest_uuid: "4c8da1d580614449a636b855bf11b95f"
from {username='slave'} at 127.0.0.1:59136
I20240501 20:01:51.933869 29652 raft_consensus.cc:482] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:01:51.899765 31433 raft_consensus.cc:686] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [term 7 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:51.934636 31425 raft_consensus.cc:3032] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [term 5 LEADER]: Stepping down as leader of term 5
I20240501 20:01:51.935034 31425 raft_consensus.cc:729] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:51.935067 31433 consensus_queue.cc:237] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:51.935899 31425 consensus_queue.cc:260] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:51.937017 31425 raft_consensus.cc:3037] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:51.938333 31268 consensus_queue.cc:1046] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:51.934257 29652 raft_consensus.cc:3037] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:51.949070 31307 raft_consensus.cc:504] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:51.949932 29652 raft_consensus.cc:504] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:51.952978 29652 leader_election.cc:288] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:51.953599 29651 raft_consensus.cc:1261] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240501 20:01:51.951237 31307 leader_election.cc:288] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 7 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:51.969120 31506 raft_consensus.cc:3032] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 6 LEADER]: Stepping down as leader of term 6
I20240501 20:01:51.969563 31506 raft_consensus.cc:729] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:51.970338 31506 consensus_queue.cc:260] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:51.971503 31506 raft_consensus.cc:3037] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Advancing to term 7
W20240501 20:01:51.964751 31713 log.cc:924] Time spent T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
W20240501 20:01:51.990670 31697 log.cc:924] Time spent T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 0.179s user 0.000s sys 0.003s
W20240501 20:01:51.992964 31712 log.cc:924] Time spent T 00000000000000000000000000000000 P 995802798c1e438daecd368538bebecd: Append to log took a long time: real 0.097s user 0.001s sys 0.000s
W20240501 20:01:51.973505 31715 log.cc:924] Time spent T a8e9f75387564275a60d38e9a910319c P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 0.089s user 0.000s sys 0.001s
W20240501 20:01:51.996702 29606 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica 4c8da1d580614449a636b855bf11b95f is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } }
I20240501 20:01:51.997220 29659 raft_consensus.cc:2454] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 5.
I20240501 20:01:51.998291 29808 raft_consensus.cc:2454] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 7.
I20240501 20:01:52.003124 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5525f546426c47a38edf7f56ad04b7ab" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:52.006640 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e49dd5180f04313ab33942fb3ddc147" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:52.007373 29658 raft_consensus.cc:3037] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:01:52.003870 29809 raft_consensus.cc:3032] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [term 5 LEADER]: Stepping down as leader of term 5
I20240501 20:01:52.007934 29809 raft_consensus.cc:729] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:52.016355 29772 leader_election.cc:302] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:52.008917 29809 consensus_queue.cc:260] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:52.025228 29809 raft_consensus.cc:3037] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Advancing to term 6
W20240501 20:01:52.031770 29606 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 3c1a08b455b94ae089396a15c837ddc2, TS fdfeac4190e045f290e003b2298d988d -> TS 888d99522e4844118f8f1cf26f268c8c move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:52.032215 29606 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 3c1a08b455b94ae089396a15c837ddc2, TS fdfeac4190e045f290e003b2298d988d -> TS 888d99522e4844118f8f1cf26f268c8c move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:51.997731 31493 raft_consensus.cc:2726] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861) with higher term. Message: Invalid argument: T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f for earlier term 4. Current term is 5.
I20240501 20:01:52.048910 29658 raft_consensus.cc:2454] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 7.
I20240501 20:01:52.061388 29772 leader_election.cc:302] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:52.071288 29809 raft_consensus.cc:2454] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 6.
I20240501 20:01:52.074045 29625 leader_election.cc:302] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
W20240501 20:01:52.076207 29606 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 69915dd5992641e3a24f7f45f1cd5c0c, TS 4c8da1d580614449a636b855bf11b95f -> TS 888d99522e4844118f8f1cf26f268c8c move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } }
W20240501 20:01:52.076532 29606 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 69915dd5992641e3a24f7f45f1cd5c0c, TS 4c8da1d580614449a636b855bf11b95f -> TS 888d99522e4844118f8f1cf26f268c8c move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } }
I20240501 20:01:52.068509 31445 raft_consensus.cc:2781] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:01:52.081924 31501 raft_consensus.cc:482] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 888d99522e4844118f8f1cf26f268c8c)
I20240501 20:01:52.082688 31501 raft_consensus.cc:504] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:52.077589 31063 raft_consensus.cc:2781] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Leader election won for term 6
I20240501 20:01:52.085255 31501 leader_election.cc:288] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227)
I20240501 20:01:52.070910 31634 raft_consensus.cc:2781] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 7 FOLLOWER]: Leader election won for term 7
I20240501 20:01:52.088006 31634 raft_consensus.cc:686] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 7 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:52.088798 31634 consensus_queue.cc:237] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:52.092435 30417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05df43bdfa479b902b1da297fb5974" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "888d99522e4844118f8f1cf26f268c8c" is_pre_election: true
I20240501 20:01:52.107141 31445 raft_consensus.cc:686] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [term 5 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:52.108156 31445 consensus_queue.cc:237] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:52.116484 29658 raft_consensus.cc:1261] T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f [term 7 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240501 20:01:52.120666 31135 consensus_queue.cc:1035] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20240501 20:01:52.123082 31434 raft_consensus.cc:3032] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 4 LEADER]: Stepping down as leader of term 4
I20240501 20:01:52.123469 31434 raft_consensus.cc:729] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:52.124310 31434 consensus_queue.cc:260] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:52.125540 31434 raft_consensus.cc:3037] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Advancing to term 5
W20240501 20:01:52.138324 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 408558us
I20240501 20:01:52.139153 29698 leader_election.cc:302] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:52.142712 29698 leader_election.cc:302] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:52.144253 31488 raft_consensus.cc:2726] T 2d86f8848a184a87947fd927c44e5a35 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240501 20:01:52.144894 31488 raft_consensus.cc:2726] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 20:01:52.145960 31488 consensus_queue.cc:1035] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:52.149760 31658 consensus_queue.cc:1035] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:01:52.150394 31678 raft_consensus.cc:2726] T 456035a1cb3f4f7da52d0fc1dd1c656c P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240501 20:01:52.153136 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 345256us
I20240501 20:01:52.155067 29729 raft_consensus.cc:1261] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20240501 20:01:52.157454 29726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f261d2a05bf4c838532049601694b4a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:52.158035 29726 raft_consensus.cc:2454] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 5.
I20240501 20:01:52.158135 31606 raft_consensus.cc:1067] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [term 3 LEADER]: Signalling peer ee0d97247d394a8facfa774f15f3a67d to start an election
I20240501 20:01:52.143783 29698 leader_election.cc:302] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:52.160185 29805 raft_consensus.cc:1261] T 668b4a4d782c4278932dd8413dc3ab3e P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20240501 20:01:52.164222 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5525f546426c47a38edf7f56ad04b7ab" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:52.164793 29728 raft_consensus.cc:3037] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:52.174011 29698 leader_election.cc:302] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:52.178679 29728 raft_consensus.cc:2454] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 6.
I20240501 20:01:52.183454 29698 leader_election.cc:302] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:52.180271 31686 raft_consensus.cc:2726] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20240501 20:01:52.185115 29625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 95652.5us
I20240501 20:01:52.183210 31063 raft_consensus.cc:686] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [term 6 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:52.194149 29698 leader_election.cc:302] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
W20240501 20:01:52.197497 29623 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.196:37227, user_credentials={real_user=slave}} blocked reactor thread for 78692.2us
I20240501 20:01:52.201392 31267 raft_consensus.cc:2932] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 2 to 6, 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:52.208555 29698 leader_election.cc:302] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:52.185703 31640 raft_consensus.cc:3037] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:52.209641 31605 raft_consensus.cc:2726] T f6d88139f89c47ff84cad674147aae0f P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:52.210199 31605 raft_consensus.cc:2726] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240501 20:01:52.205660 31063 consensus_queue.cc:237] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:52.213131 29624 leader_election.cc:302] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:52.207993 29801 raft_consensus.cc:2932] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 2 to 6, 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:52.224953 29800 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3af08384057a4d85977481472f460168"
dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
from {username='slave'} at 127.0.0.1:41864
I20240501 20:01:52.225497 29800 raft_consensus.cc:482] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:01:52.225808 29800 raft_consensus.cc:3037] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:52.193360 31606 raft_consensus.cc:2726] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240501 20:01:52.234792 31447 raft_consensus.cc:2781] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240501 20:01:52.224707 31449 raft_consensus.cc:482] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:52.244010 31536 consensus_queue.cc:1035] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20240501 20:01:52.232486 31605 consensus_queue.cc:1035] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
I20240501 20:01:52.248580 31447 raft_consensus.cc:482] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting leader election (detected failure of leader fdfeac4190e045f290e003b2298d988d)
I20240501 20:01:52.262413 31447 raft_consensus.cc:3037] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Advancing to term 5
W20240501 20:01:52.264624 29606 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet d48fd0a8e289401aa7417c42ff2c81b2, TS fdfeac4190e045f290e003b2298d988d -> TS 888d99522e4844118f8f1cf26f268c8c move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:52.264950 29606 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet d48fd0a8e289401aa7417c42ff2c81b2, TS fdfeac4190e045f290e003b2298d988d -> TS 888d99522e4844118f8f1cf26f268c8c move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:52.250824 31449 raft_consensus.cc:504] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:52.276681 29606 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 0fb27193abbf4644a48662909bc7b04a, TS ee0d97247d394a8facfa774f15f3a67d -> TS 888d99522e4844118f8f1cf26f268c8c move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:52.277110 29606 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 0fb27193abbf4644a48662909bc7b04a, TS ee0d97247d394a8facfa774f15f3a67d -> TS 888d99522e4844118f8f1cf26f268c8c move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:52.276546 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48630ae6eb394ac1b54955dd01cf0ed2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:52.280007 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48630ae6eb394ac1b54955dd01cf0ed2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:52.281414 29624 leader_election.cc:302] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f; no voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:52.282023 29800 raft_consensus.cc:504] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:52.283919 29800 leader_election.cc:288] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:52.279310 29735 raft_consensus.cc:2932] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 2 to 6, 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:52.286674 29654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af08384057a4d85977481472f460168" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:52.287221 29654 raft_consensus.cc:3037] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Advancing to term 4
W20240501 20:01:52.287165 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 64906us
I20240501 20:01:52.275337 31449 leader_election.cc:288] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:52.291622 29554 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:43384 (request call id 145) took 4891 ms. Trace:
I20240501 20:01:52.291020 29557 catalog_manager.cc:5484] T cb05df43bdfa479b902b1da297fb5974 P 888d99522e4844118f8f1cf26f268c8c reported cstate change: term changed from 5 to 6, leader changed from fdfeac4190e045f290e003b2298d988d (127.27.217.194) to 888d99522e4844118f8f1cf26f268c8c (127.27.217.196). New cstate: current_term: 6 leader_uuid: "888d99522e4844118f8f1cf26f268c8c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 20:01:52.300068 31724 log.cc:924] Time spent T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
I20240501 20:01:52.294795 29557 catalog_manager.cc:5484] T fa7e89c0572e4ee7be8e5315b459bdb3 P 888d99522e4844118f8f1cf26f268c8c reported cstate change: term changed from 4 to 7, leader changed from 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 7 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:52.302176 29557 catalog_manager.cc:5484] T be383be783234855878cb8fd442c7b05 P 888d99522e4844118f8f1cf26f268c8c reported cstate change: term changed from 3 to 4, leader changed from ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 4 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
I20240501 20:01:52.304885 29557 catalog_manager.cc:5484] T 696121417ea9403990456cfc6fc4f416 P 888d99522e4844118f8f1cf26f268c8c reported cstate change: term changed from 3 to 5, leader changed from fdfeac4190e045f290e003b2298d988d (127.27.217.194) to 888d99522e4844118f8f1cf26f268c8c (127.27.217.196). New cstate: current_term: 5 leader_uuid: "888d99522e4844118f8f1cf26f268c8c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:01:52.306784 29557 catalog_manager.cc:5484] T 8bd5e83faaff4e2da4e4cc22ba410d33 P 888d99522e4844118f8f1cf26f268c8c reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "888d99522e4844118f8f1cf26f268c8c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:01:52.309839 31640 raft_consensus.cc:2726] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240501 20:01:52.310811 29557 catalog_manager.cc:5484] T 25e07220db7444ae97f294d70eda6a93 P 888d99522e4844118f8f1cf26f268c8c reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "888d99522e4844118f8f1cf26f268c8c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:01:52.287549 31511 raft_consensus.cc:2726] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 20:01:52.295805 29554 rpcz_store.cc:276] 0501 20:01:47.400164 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:47.400307 (+ 143us) service_pool.cc:225] Handling call
0501 20:01:49.330804 (+1930497us) spinlock_profiling.cc:243] Waited 191 ms on lock 0x7b54000037d4. stack: 00007fe45491eba3 00007fe45491eaa0 00007fe454283584 00007fe45df999e4 00007fe45df99979 00007fe457800d30 00007fe45788417a 00007fe45d7d5b1c 00007fe45d8cfa8e 00007fe45bbe3a44 00007fe45bbe39d2 00007fe45bbe3901 00007fe45bbe387c 00007fe45bbe2b32 00007fe455ca2bcc 00007fe455ca2006
0501 20:01:52.291571 (+2960767us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0501 20:01:51.800325 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 20:01:51.829039 (+ 28714us) write_op.cc:424] Acquiring schema lock in shared mode
0501 20:01:51.829067 (+ 28us) write_op.cc:427] Acquired schema lock
0501 20:01:51.829081 (+ 14us) tablet.cc:660] Decoding operations
0501 20:01:51.829849 (+ 768us) write_op.cc:612] Acquiring the partition lock for write op
0501 20:01:51.829918 (+ 69us) write_op.cc:633] Partition lock acquired for write op
0501 20:01:51.829941 (+ 23us) tablet.cc:683] Acquiring locks for 9 operations
0501 20:01:51.830536 (+ 595us) tablet.cc:699] Row locks acquired
0501 20:01:51.830553 (+ 17us) write_op.cc:260] PREPARE: finished
0501 20:01:51.830677 (+ 124us) write_op.cc:270] Start()
0501 20:01:51.830774 (+ 97us) write_op.cc:276] Timestamp: P: 1714593711830638 usec, L: 0
0501 20:01:51.830793 (+ 19us) op_driver.cc:331] REPLICATION: starting
0501 20:01:51.831316 (+ 523us) log.cc:842] Serialized 3672 byte log entry
0501 20:01:52.074232 (+242916us) op_driver.cc:444] REPLICATION: finished
0501 20:01:52.119887 (+ 45655us) write_op.cc:301] APPLY: starting
0501 20:01:52.120001 (+ 114us) tablet.cc:1365] starting BulkCheckPresence
0501 20:01:52.124858 (+ 4857us) tablet.cc:1368] finished BulkCheckPresence
0501 20:01:52.125029 (+ 171us) tablet.cc:1370] starting ApplyRowOperation cycle
0501 20:01:52.126095 (+ 1066us) tablet.cc:1381] finished ApplyRowOperation cycle
0501 20:01:52.126384 (+ 289us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=9
0501 20:01:52.126445 (+ 61us) write_op.cc:312] APPLY: finished
0501 20:01:52.126824 (+ 379us) log.cc:842] Serialized 70 byte log entry
0501 20:01:52.128211 (+ 1387us) write_op.cc:481] Releasing partition, row and schema locks
0501 20:01:52.128674 (+ 463us) write_op.cc:446] Released schema lock
0501 20:01:52.129109 (+ 435us) write_op.cc:341] FINISH: Updating metrics
0501 20:01:52.141801 (+ 12692us) write_op.cc:481] Releasing partition, row and schema locks
0501 20:01:52.141854 (+ 53us) write_op.cc:446] Released schema lock
Metrics: {"mutex_wait_us":316750,"reported_tablets":43,"spinlock_wait_cycles":440369536,"child_traces":[["op",{"apply.queue_time_us":45363,"apply.run_cpu_time_us":3588,"apply.run_wall_time_us":22309,"num_ops":9,"prepare.queue_time_us":198531,"prepare.run_cpu_time_us":14761,"prepare.run_wall_time_us":111057,"raft.queue_time_us":79586,"raft.run_cpu_time_us":28627,"raft.run_wall_time_us":44036,"replication_time_us":243354,"thread_start_us":324408,"threads_started":4,"wal-append.queue_time_us":64189}]]}
W20240501 20:01:52.330775 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 92276.9us
I20240501 20:01:52.336127 29655 raft_consensus.cc:1261] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20240501 20:01:52.336575 29655 pending_rounds.cc:76] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f: Aborting all ops after (but not including) 2
I20240501 20:01:52.336838 29655 pending_rounds.cc:97] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20240501 20:01:52.337114 29655 raft_consensus.cc:2866] T 668b4a4d782c4278932dd8413dc3ab3e P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 20:01:52.337227 29654 raft_consensus.cc:2454] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 4.
I20240501 20:01:52.338984 29772 leader_election.cc:302] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:52.343065 29652 raft_consensus.cc:1226] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [term 7 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 6. Current term is 7. Ops: []
I20240501 20:01:52.340808 31467 raft_consensus.cc:2781] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:01:52.356575 31467 raft_consensus.cc:686] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [term 4 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:52.360149 31467 consensus_queue.cc:237] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:52.363550 29801 raft_consensus.cc:1226] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 7 LEADER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 6. Current term is 7. Ops: []
I20240501 20:01:52.373143 29735 raft_consensus.cc:1226] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 7 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 5. Current term is 7. Ops: []
I20240501 20:01:52.373577 31606 consensus_queue.cc:1046] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:52.374375 31606 consensus_queue.cc:1225] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [LEADER]: Peer ee0d97247d394a8facfa774f15f3a67d log is divergent from this leader: its last log entry 7.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20240501 20:01:52.374495 29801 raft_consensus.cc:1261] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 7 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240501 20:01:52.374888 31606 raft_consensus.cc:3032] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 6 LEADER]: Stepping down as leader of term 6
I20240501 20:01:52.375262 31606 raft_consensus.cc:729] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:52.376319 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9472c53f79ec423f82c396bd980674e2" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:52.376865 29732 raft_consensus.cc:3037] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:52.377252 29734 raft_consensus.cc:1226] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 5. Current term is 6. Ops: []
I20240501 20:01:52.378547 31153 consensus_queue.cc:1035] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:01:52.380394 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07bdb8b09b8c4ef4b8e49daa9b8c01d6" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:52.380918 29729 raft_consensus.cc:3037] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:01:52.381580 29734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e49dd5180f04313ab33942fb3ddc147" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:52.384860 29726 raft_consensus.cc:1261] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d [term 7 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240501 20:01:52.385345 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb05df43bdfa479b902b1da297fb5974" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:52.375960 31606 consensus_queue.cc:260] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:52.388675 29733 raft_consensus.cc:1261] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 7 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240501 20:01:52.390103 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3af08384057a4d85977481472f460168" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:52.390590 29733 raft_consensus.cc:3032] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:01:52.390852 31267 consensus_queue.cc:1035] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240501 20:01:52.390852 29733 raft_consensus.cc:729] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:52.391115 31606 raft_consensus.cc:3037] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:01:52.396677 29733 consensus_queue.cc:260] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:52.400789 29773 leader_election.cc:302] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d; no voters: 888d99522e4844118f8f1cf26f268c8c, fdfeac4190e045f290e003b2298d988d
I20240501 20:01:52.400651 31307 consensus_queue.cc:1035] T d48fd0a8e289401aa7417c42ff2c81b2 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240501 20:01:52.401679 31135 raft_consensus.cc:2726] T cb05df43bdfa479b902b1da297fb5974 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 20:01:52.406284 31691 consensus_queue.cc:1035] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 20:01:52.296641 30417 raft_consensus.cc:2932] T be383be783234855878cb8fd442c7b05 P 888d99522e4844118f8f1cf26f268c8c [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 2 to 6, 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:52.422348 29729 raft_consensus.cc:2454] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 7.
I20240501 20:01:52.432932 29732 raft_consensus.cc:2454] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 5.
W20240501 20:01:52.434737 31731 log.cc:924] Time spent T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 0.063s user 0.001s sys 0.000s
W20240501 20:01:52.436789 31725 log.cc:924] Time spent T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d: Append to log took a long time: real 0.179s user 0.000s sys 0.003s
I20240501 20:01:52.452781 29773 leader_election.cc:302] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters: 4c8da1d580614449a636b855bf11b95f
W20240501 20:01:52.477031 31726 log.cc:924] Time spent T d48fd0a8e289401aa7417c42ff2c81b2 P 4c8da1d580614449a636b855bf11b95f: Append to log took a long time: real 0.215s user 0.002s sys 0.002s
I20240501 20:01:52.461663 31445 raft_consensus.cc:2781] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [term 7 FOLLOWER]: Leader election won for term 7
I20240501 20:01:52.531374 29653 raft_consensus.cc:1226] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Rejecting Update request from peer fdfeac4190e045f290e003b2298d988d for earlier term 3. Current term is 4. Ops: []
I20240501 20:01:52.531414 29733 raft_consensus.cc:3037] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:52.540772 29558 catalog_manager.cc:5484] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 4 to 5, leader changed from fdfeac4190e045f290e003b2298d988d (127.27.217.194) to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 5 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:52.448925 31447 raft_consensus.cc:504] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:52.560431 29651 raft_consensus.cc:1261] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240501 20:01:52.562178 29727 raft_consensus.cc:1261] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240501 20:01:52.563854 29653 raft_consensus.cc:1261] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [term 7 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240501 20:01:52.577309 29733 raft_consensus.cc:2454] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 4.
I20240501 20:01:52.592813 31447 leader_election.cc:288] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:52.596058 29734 raft_consensus.cc:2396] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ee0d97247d394a8facfa774f15f3a67d for term 7 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20240501 20:01:52.598590 29729 raft_consensus.cc:1261] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 7 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240501 20:01:52.604319 31501 consensus_queue.cc:1035] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:52.606751 31467 consensus_queue.cc:1035] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:52.607354 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f261d2a05bf4c838532049601694b4a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:52.608068 29809 raft_consensus.cc:2379] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 5: Already voted for candidate ee0d97247d394a8facfa774f15f3a67d in this term.
I20240501 20:01:52.555059 31445 raft_consensus.cc:686] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [term 7 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:52.613476 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f261d2a05bf4c838532049601694b4a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:52.562371 29558 catalog_manager.cc:5484] T 8edcd903224d48f889b46e1b5318e0ec P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 2 to 5, leader changed from ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 5 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } }
I20240501 20:01:52.618947 29558 catalog_manager.cc:5484] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 3 to 6, leader changed from fdfeac4190e045f290e003b2298d988d (127.27.217.194) to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 6 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:52.625069 29727 raft_consensus.cc:2454] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 5.
I20240501 20:01:52.628909 29624 leader_election.cc:302] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters: ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:52.635782 29558 catalog_manager.cc:5484] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 1 to 5. New cstate: current_term: 5 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:52.638419 29558 catalog_manager.cc:5484] T bfac893b4b4f457694a4eb184b6abb53 P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 2 to 5. New cstate: current_term: 5 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:52.641213 29558 catalog_manager.cc:5484] T 77a35893b91247a99912eba787d787cf P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 3 to 4, leader changed from fdfeac4190e045f290e003b2298d988d (127.27.217.194) to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 4 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:52.644584 29558 catalog_manager.cc:5484] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 3 to 4, leader changed from 4c8da1d580614449a636b855bf11b95f (127.27.217.193) to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 4 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } }
I20240501 20:01:52.647888 29558 catalog_manager.cc:5484] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 4 to 5, leader changed from ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 5 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } }
I20240501 20:01:52.650720 29558 catalog_manager.cc:5484] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:52.652284 29558 catalog_manager.cc:5484] T a8e9f75387564275a60d38e9a910319c P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 2 to 3, leader changed from 4c8da1d580614449a636b855bf11b95f (127.27.217.193) to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 3 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } }
I20240501 20:01:52.653570 29558 catalog_manager.cc:5484] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 1 to 4, leader changed from ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 4 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } }
I20240501 20:01:52.655326 29558 catalog_manager.cc:5484] T 49da4246f5f6438abe0e7e8c9a523d80 P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 6 to 7, leader changed from fdfeac4190e045f290e003b2298d988d (127.27.217.194) to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 7 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:52.656960 29558 catalog_manager.cc:5484] T 3c1a08b455b94ae089396a15c837ddc2 P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 2 to 4, leader changed from 4c8da1d580614449a636b855bf11b95f (127.27.217.193) to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 4 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:52.658318 29558 catalog_manager.cc:5484] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 4 to 7, leader changed from fdfeac4190e045f290e003b2298d988d (127.27.217.194) to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 7 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:52.637619 31447 raft_consensus.cc:2781] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:01:52.659762 29558 catalog_manager.cc:5484] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 3 to 5, leader changed from 4c8da1d580614449a636b855bf11b95f (127.27.217.193) to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 5 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:52.608085 31467 consensus_queue.cc:1035] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.039s
I20240501 20:01:52.662726 29558 catalog_manager.cc:5484] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 1 to 2, leader changed from 4c8da1d580614449a636b855bf11b95f (127.27.217.193) to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 2 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } }
I20240501 20:01:52.665123 29558 catalog_manager.cc:5484] T c651044a305c4168805825a53d910899 P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 3 to 4, leader changed from ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 4 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } }
I20240501 20:01:52.666571 29558 catalog_manager.cc:5484] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 4 to 5, leader changed from fdfeac4190e045f290e003b2298d988d (127.27.217.194) to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 5 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:52.668994 29558 catalog_manager.cc:5484] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 3 to 4, leader changed from 4c8da1d580614449a636b855bf11b95f (127.27.217.193) to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 4 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:52.674430 31648 raft_consensus.cc:482] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:52.674991 31648 raft_consensus.cc:504] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:52.676990 31648 leader_election.cc:288] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:52.679749 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649b71f9f267437ca51c758dd9ca6987" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:52.680464 29660 raft_consensus.cc:2454] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 6.
I20240501 20:01:52.648828 31472 consensus_queue.cc:1035] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.032s
I20240501 20:01:52.680922 31472 consensus_queue.cc:1225] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Peer fdfeac4190e045f290e003b2298d988d log is divergent from this leader: its last log entry 6.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20240501 20:01:52.681763 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649b71f9f267437ca51c758dd9ca6987" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:52.681798 29772 leader_election.cc:302] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:52.686589 29728 pending_rounds.cc:76] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d: Aborting all ops after (but not including) 2
I20240501 20:01:52.686879 29728 pending_rounds.cc:97] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d: Aborting uncommitted NO_OP operation due to leader change: 6.3
I20240501 20:01:52.687114 29728 raft_consensus.cc:2866] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 20:01:52.687376 31470 raft_consensus.cc:2781] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240501 20:01:52.687731 31470 raft_consensus.cc:482] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:52.688074 31470 raft_consensus.cc:3037] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:01:52.697496 31447 raft_consensus.cc:686] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [term 5 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:52.698529 31447 consensus_queue.cc:237] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:52.661448 31467 consensus_queue.cc:1225] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Peer fdfeac4190e045f290e003b2298d988d log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20240501 20:01:52.705325 31445 consensus_queue.cc:237] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:52.717334 29729 pending_rounds.cc:76] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d: Aborting all ops after (but not including) 1
I20240501 20:01:52.725733 29729 pending_rounds.cc:97] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20240501 20:01:52.738222 29729 raft_consensus.cc:2866] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 20:01:52.740653 31470 raft_consensus.cc:504] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:52.742482 29733 raft_consensus.cc:1261] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240501 20:01:52.773993 29557 catalog_manager.cc:5484] T be383be783234855878cb8fd442c7b05 P 888d99522e4844118f8f1cf26f268c8c reported cstate change: config changed from index 2 to 6, 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:52.805488 31470 leader_election.cc:288] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 7 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:52.805318 31447 consensus_queue.cc:1035] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:01:52.807405 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649b71f9f267437ca51c758dd9ca6987" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:52.811820 29652 raft_consensus.cc:3037] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:01:52.815410 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649b71f9f267437ca51c758dd9ca6987" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:52.816744 29733 raft_consensus.cc:3037] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:01:52.799855 29558 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:43396 (request call id 163) took 2831 ms. Trace:
I20240501 20:01:52.825228 29733 raft_consensus.cc:2454] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 7.
I20240501 20:01:52.824463 29558 rpcz_store.cc:276] 0501 20:01:49.968036 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:01:49.978275 (+ 10239us) service_pool.cc:225] Handling call
0501 20:01:52.692577 (+2714302us) write_op.cc:481] Releasing partition, row and schema locks
0501 20:01:52.692613 (+ 36us) write_op.cc:446] Released schema lock
0501 20:01:52.799787 (+107174us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0501 20:01:52.679142 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 20:01:52.679602 (+ 460us) write_op.cc:424] Acquiring schema lock in shared mode
0501 20:01:52.679645 (+ 43us) write_op.cc:427] Acquired schema lock
0501 20:01:52.679658 (+ 13us) tablet.cc:660] Decoding operations
0501 20:01:52.681163 (+ 1505us) write_op.cc:612] Acquiring the partition lock for write op
0501 20:01:52.681216 (+ 53us) write_op.cc:633] Partition lock acquired for write op
0501 20:01:52.681244 (+ 28us) tablet.cc:683] Acquiring locks for 19 operations
0501 20:01:52.682895 (+ 1651us) tablet.cc:699] Row locks acquired
0501 20:01:52.682918 (+ 23us) write_op.cc:260] PREPARE: finished
0501 20:01:52.683114 (+ 196us) write_op.cc:270] Start()
0501 20:01:52.683245 (+ 131us) write_op.cc:276] Timestamp: P: 1714593712683063 usec, L: 0
0501 20:01:52.683268 (+ 23us) op_driver.cc:331] REPLICATION: starting
0501 20:01:52.684013 (+ 745us) log.cc:842] Serialized 7582 byte log entry
0501 20:01:52.687382 (+ 3369us) op_driver.cc:444] REPLICATION: finished
0501 20:01:52.687910 (+ 528us) write_op.cc:301] APPLY: starting
0501 20:01:52.688003 (+ 93us) tablet.cc:1365] starting BulkCheckPresence
0501 20:01:52.688469 (+ 466us) tablet.cc:1368] finished BulkCheckPresence
0501 20:01:52.688587 (+ 118us) tablet.cc:1370] starting ApplyRowOperation cycle
0501 20:01:52.690290 (+ 1703us) tablet.cc:1381] finished ApplyRowOperation cycle
0501 20:01:52.690536 (+ 246us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=19
0501 20:01:52.690578 (+ 42us) write_op.cc:312] APPLY: finished
0501 20:01:52.690901 (+ 323us) log.cc:842] Serialized 131 byte log entry
0501 20:01:52.691057 (+ 156us) write_op.cc:481] Releasing partition, row and schema locks
0501 20:01:52.691220 (+ 163us) write_op.cc:446] Released schema lock
0501 20:01:52.691285 (+ 65us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":1459,"reported_tablets":52,"child_traces":[["op",{"apply.queue_time_us":281,"apply.run_cpu_time_us":3641,"apply.run_wall_time_us":4394,"num_ops":19,"prepare.queue_time_us":117,"prepare.run_cpu_time_us":5198,"prepare.run_wall_time_us":13336,"replication_time_us":4007}]]}
I20240501 20:01:52.829074 29652 raft_consensus.cc:2454] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 7.
I20240501 20:01:52.830219 29772 leader_election.cc:302] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:52.830989 31470 raft_consensus.cc:2781] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [term 7 FOLLOWER]: Leader election won for term 7
I20240501 20:01:52.831437 31470 raft_consensus.cc:686] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [term 7 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:52.832190 31470 consensus_queue.cc:237] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:52.850667 29729 raft_consensus.cc:1261] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240501 20:01:52.852308 31445 consensus_queue.cc:1035] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:52.877382 29556 catalog_manager.cc:5484] T bbcf30cf66714efdac997862dafbe4a2 P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 5 to 6, leader changed from 4c8da1d580614449a636b855bf11b95f (127.27.217.193) to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 6 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:52.880246 29556 catalog_manager.cc:5484] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:52.881918 29652 raft_consensus.cc:1261] T 3af08384057a4d85977481472f460168 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240501 20:01:52.882532 29556 catalog_manager.cc:5484] T 668b4a4d782c4278932dd8413dc3ab3e P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 3 to 6. New cstate: current_term: 6 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:52.883252 31639 consensus_queue.cc:1035] T 3af08384057a4d85977481472f460168 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:52.884425 29556 catalog_manager.cc:5484] T 36057b3f24d44b9aac3a1378053a395f P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 3 to 4, leader changed from 4c8da1d580614449a636b855bf11b95f (127.27.217.193) to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 4 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:52.887413 29556 catalog_manager.cc:5484] T 69915dd5992641e3a24f7f45f1cd5c0c P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 4 to 6, leader changed from ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 6 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:52.889540 29556 catalog_manager.cc:5484] T a8787bc4d6b44649b9138d906730110a P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 3 to 4, leader changed from 4c8da1d580614449a636b855bf11b95f (127.27.217.193) to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 4 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:52.891505 29556 catalog_manager.cc:5484] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 2 to 3, leader changed from 4c8da1d580614449a636b855bf11b95f (127.27.217.193) to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 3 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:52.893019 29809 raft_consensus.cc:1261] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240501 20:01:52.893509 29556 catalog_manager.cc:5484] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 2 to 4, leader changed from 4c8da1d580614449a636b855bf11b95f (127.27.217.193) to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 4 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:52.896189 29556 catalog_manager.cc:5484] T 6e38a2e91a3d48d09a22abce723af833 P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 2 to 5, leader changed from ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 5 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } }
I20240501 20:01:52.896951 31447 consensus_queue.cc:1035] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:01:52.973814 31447 raft_consensus.cc:482] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:52.974704 31447 raft_consensus.cc:504] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:52.978536 31447 leader_election.cc:288] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:52.984114 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece8d4ed84c34b748f7b6272b31c90ac" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:52.984791 29809 raft_consensus.cc:2454] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 6.
I20240501 20:01:52.985836 29625 leader_election.cc:302] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:52.986363 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece8d4ed84c34b748f7b6272b31c90ac" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:52.986989 31268 raft_consensus.cc:2781] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240501 20:01:52.987397 31268 raft_consensus.cc:482] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting leader election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:52.987757 31268 raft_consensus.cc:3037] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:52.987007 29728 raft_consensus.cc:2379] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 6: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:52.995079 31268 raft_consensus.cc:504] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:52.997712 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece8d4ed84c34b748f7b6272b31c90ac" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:52.998441 29728 raft_consensus.cc:2379] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 6: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:52.999975 31268 leader_election.cc:288] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:53.000840 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece8d4ed84c34b748f7b6272b31c90ac" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:53.007427 29809 raft_consensus.cc:2454] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 6.
I20240501 20:01:53.010121 29625 leader_election.cc:302] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters: fdfeac4190e045f290e003b2298d988d
I20240501 20:01:53.007267 29554 catalog_manager.cc:5484] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 5 to 7, leader changed from ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 7 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:53.012223 31268 raft_consensus.cc:2781] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Leader election won for term 6
I20240501 20:01:53.013957 31268 raft_consensus.cc:686] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [term 6 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:53.016593 31268 consensus_queue.cc:237] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:53.019537 29554 catalog_manager.cc:5484] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 5 to 6, leader changed from ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 6 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:53.039135 29556 catalog_manager.cc:5484] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 1 to 5, leader changed from 4c8da1d580614449a636b855bf11b95f (127.27.217.193) to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 5 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } }
I20240501 20:01:53.044075 29556 catalog_manager.cc:5484] T 3af08384057a4d85977481472f460168 P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 3 to 4, leader changed from fdfeac4190e045f290e003b2298d988d (127.27.217.194) to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 4 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } }
I20240501 20:01:53.068634 29556 catalog_manager.cc:5484] T d48fd0a8e289401aa7417c42ff2c81b2 P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 5 to 7, leader changed from 4c8da1d580614449a636b855bf11b95f (127.27.217.193) to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 7 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } }
I20240501 20:01:53.071487 29556 catalog_manager.cc:5484] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 4 to 7, leader changed from fdfeac4190e045f290e003b2298d988d (127.27.217.194) to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 7 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } }
I20240501 20:01:53.087024 31447 raft_consensus.cc:482] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:53.087486 31447 raft_consensus.cc:504] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:53.091135 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5525fc4b82a44b0991c3dd6098fde81" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:53.091684 29809 raft_consensus.cc:2454] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 6.
I20240501 20:01:53.092223 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5525fc4b82a44b0991c3dd6098fde81" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:53.092983 29728 raft_consensus.cc:2379] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 6: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:53.095515 29625 leader_election.cc:302] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters: fdfeac4190e045f290e003b2298d988d
I20240501 20:01:53.089248 31447 leader_election.cc:288] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:53.096730 31426 raft_consensus.cc:2781] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240501 20:01:53.097103 31426 raft_consensus.cc:482] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting leader election (detected failure of leader ee0d97247d394a8facfa774f15f3a67d)
I20240501 20:01:53.097455 31426 raft_consensus.cc:3037] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:53.099303 29652 raft_consensus.cc:1261] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 7 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 3. (term mismatch)
I20240501 20:01:53.099647 29652 pending_rounds.cc:76] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f: Aborting all ops after (but not including) 2
I20240501 20:01:53.099857 29652 pending_rounds.cc:97] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f: Aborting uncommitted NO_OP operation due to leader change: 6.3
I20240501 20:01:53.100097 29652 raft_consensus.cc:2866] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P 4c8da1d580614449a636b855bf11b95f [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 20:01:53.102041 31501 consensus_queue.cc:1035] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:53.108091 31426 raft_consensus.cc:504] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:53.112313 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5525fc4b82a44b0991c3dd6098fde81" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:53.110095 31426 leader_election.cc:288] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:53.113039 29728 raft_consensus.cc:2379] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4c8da1d580614449a636b855bf11b95f in current term 6: Already voted for candidate fdfeac4190e045f290e003b2298d988d in this term.
I20240501 20:01:53.115370 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5525fc4b82a44b0991c3dd6098fde81" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:53.118163 29557 catalog_manager.cc:5484] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:53.122306 29809 raft_consensus.cc:2454] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 6.
I20240501 20:01:53.124281 29625 leader_election.cc:302] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters: fdfeac4190e045f290e003b2298d988d
I20240501 20:01:53.125000 31426 raft_consensus.cc:2781] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Leader election won for term 6
I20240501 20:01:53.125537 31426 raft_consensus.cc:686] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [term 6 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:53.125589 29728 raft_consensus.cc:1261] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P fdfeac4190e045f290e003b2298d988d [term 7 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 7 index: 3. (index mismatch)
I20240501 20:01:53.126559 31426 consensus_queue.cc:237] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:53.127537 31445 consensus_queue.cc:1035] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:53.133785 29557 catalog_manager.cc:5484] T 07bdb8b09b8c4ef4b8e49daa9b8c01d6 P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 4 to 7, leader changed from fdfeac4190e045f290e003b2298d988d (127.27.217.194) to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 7 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:53.165112 29652 consensus_queue.cc:237] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } }
I20240501 20:01:53.170621 29809 raft_consensus.cc:1261] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20240501 20:01:53.171612 31268 consensus_queue.cc:1035] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:01:53.174163 30417 raft_consensus.cc:1261] T be383be783234855878cb8fd442c7b05 P 888d99522e4844118f8f1cf26f268c8c [term 4 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20240501 20:01:53.175779 31447 consensus_queue.cc:1035] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:01:53.181751 31268 raft_consensus.cc:2932] T be383be783234855878cb8fd442c7b05 P 4c8da1d580614449a636b855bf11b95f [term 4 LEADER]: Committing config change with OpId 4.7: config changed from index 6 to 7, VOTER fdfeac4190e045f290e003b2298d988d (127.27.217.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:53.184624 29809 raft_consensus.cc:2932] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Committing config change with OpId 4.7: config changed from index 6 to 7, VOTER fdfeac4190e045f290e003b2298d988d (127.27.217.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:53.192458 30417 raft_consensus.cc:2932] T be383be783234855878cb8fd442c7b05 P 888d99522e4844118f8f1cf26f268c8c [term 4 FOLLOWER]: Committing config change with OpId 4.7: config changed from index 6 to 7, VOTER fdfeac4190e045f290e003b2298d988d (127.27.217.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:53.216671 29556 catalog_manager.cc:5484] T be383be783234855878cb8fd442c7b05 P ee0d97247d394a8facfa774f15f3a67d reported cstate change: config changed from index 6 to 7, VOTER fdfeac4190e045f290e003b2298d988d (127.27.217.194) evicted. New cstate: current_term: 4 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: false } } }
I20240501 20:01:53.220309 29542 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet be383be783234855878cb8fd442c7b05 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:01:53.236503 29725 tablet_service.cc:1514] Processing DeleteTablet for tablet be383be783234855878cb8fd442c7b05 with delete_type TABLET_DATA_TOMBSTONED (TS fdfeac4190e045f290e003b2298d988d not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:36258
I20240501 20:01:53.246376 31758 tablet_replica.cc:330] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d: stopping tablet replica
I20240501 20:01:53.247660 31758 raft_consensus.cc:2229] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:53.253890 31758 raft_consensus.cc:2258] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:53.253705 29556 catalog_manager.cc:5484] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 4 to 5, leader changed from fdfeac4190e045f290e003b2298d988d (127.27.217.194) to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 5 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:53.256109 29556 catalog_manager.cc:5484] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 5 to 6, leader changed from ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 6 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:53.258231 29556 catalog_manager.cc:5484] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 5 to 6, leader changed from ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 6 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:53.259845 31758 ts_tablet_manager.cc:1875] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:01:53.291332 31758 ts_tablet_manager.cc:1888] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.6
I20240501 20:01:53.291810 31758 log.cc:1196] T be383be783234855878cb8fd442c7b05 P fdfeac4190e045f290e003b2298d988d: Deleting WAL directory at /tmp/dist-test-taskOXOMEu/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1714593602638719-28519-0/minicluster-data/ts-1-root/wals/be383be783234855878cb8fd442c7b05
I20240501 20:01:53.293378 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:01:53.293431 29543 catalog_manager.cc:4830] TS fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907): tablet be383be783234855878cb8fd442c7b05 (table test-workload [id=c4cd6efa2a67402dac287d449c66135b]) successfully deleted
I20240501 20:01:53.314951 29809 raft_consensus.cc:1261] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240501 20:01:53.316717 31426 consensus_queue.cc:1035] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:01:53.323922 29728 raft_consensus.cc:1261] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240501 20:01:53.329037 31426 consensus_queue.cc:1035] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:01:53.361001 29729 raft_consensus.cc:1261] T 649b71f9f267437ca51c758dd9ca6987 P fdfeac4190e045f290e003b2298d988d [term 7 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240501 20:01:53.362756 31501 consensus_queue.cc:1035] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:01:53.359853 29651 raft_consensus.cc:1261] T 649b71f9f267437ca51c758dd9ca6987 P 4c8da1d580614449a636b855bf11b95f [term 7 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240501 20:01:53.376214 31445 consensus_queue.cc:1035] T 649b71f9f267437ca51c758dd9ca6987 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:53.428395 31445 raft_consensus.cc:482] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:53.428902 31445 raft_consensus.cc:504] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:53.430918 31445 leader_election.cc:288] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:53.435715 29651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77be7101c0fe4085b6706296f609870a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:53.436447 29651 raft_consensus.cc:2454] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 5.
I20240501 20:01:53.437273 29772 leader_election.cc:302] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:53.438374 31307 raft_consensus.cc:2781] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240501 20:01:53.438791 31307 raft_consensus.cc:482] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:53.440722 31307 raft_consensus.cc:3037] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:53.444577 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77be7101c0fe4085b6706296f609870a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:53.464438 29809 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "24a2bba62f434d1c8489b55c8fec2187"
dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
mode: GRACEFUL
new_leader_uuid: "fdfeac4190e045f290e003b2298d988d"
from {username='slave'} at 127.0.0.1:33582
I20240501 20:01:53.465067 29809 raft_consensus.cc:595] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [term 4 LEADER]: Received request to transfer leadership to TS fdfeac4190e045f290e003b2298d988d
I20240501 20:01:53.467535 29809 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4e49dd5180f04313ab33942fb3ddc147"
dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
mode: GRACEFUL
new_leader_uuid: "4c8da1d580614449a636b855bf11b95f"
from {username='slave'} at 127.0.0.1:33582
I20240501 20:01:53.468066 29809 raft_consensus.cc:595] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 7 LEADER]: Received request to transfer leadership to TS 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:53.471841 29732 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aa8cd40cb4044d18a86001ce6ddf97ab"
dest_uuid: "fdfeac4190e045f290e003b2298d988d"
mode: GRACEFUL
new_leader_uuid: "ee0d97247d394a8facfa774f15f3a67d"
from {username='slave'} at 127.0.0.1:46074
I20240501 20:01:53.472448 29732 raft_consensus.cc:595] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [term 2 LEADER]: Received request to transfer leadership to TS ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:53.471830 29728 raft_consensus.cc:1261] T ece8d4ed84c34b748f7b6272b31c90ac P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240501 20:01:53.476507 31426 consensus_queue.cc:1035] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:01:53.485132 29651 consensus_queue.cc:237] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
W20240501 20:01:53.491070 29623 consensus_peers.cc:485] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f -> Peer 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227): Couldn't send request to peer 888d99522e4844118f8f1cf26f268c8c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc312e7affec4f67ada04db849694bdc. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:53.494311 29809 raft_consensus.cc:1261] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 7 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240501 20:01:53.495828 31426 consensus_queue.cc:1035] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:01:53.504487 29801 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b337b8e3e38d4ad6b1076fbd0d0a1609"
dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
mode: GRACEFUL
new_leader_uuid: "4c8da1d580614449a636b855bf11b95f"
from {username='slave'} at 127.0.0.1:33582
I20240501 20:01:53.505092 29801 raft_consensus.cc:595] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [term 4 LEADER]: Received request to transfer leadership to TS 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:53.507539 29801 raft_consensus.cc:1261] T ece8d4ed84c34b748f7b6272b31c90ac P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240501 20:01:53.508751 31448 consensus_queue.cc:1035] T ece8d4ed84c34b748f7b6272b31c90ac P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:01:53.508882 31307 raft_consensus.cc:504] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:53.511796 29732 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d1d2cac25f354a97b1bfb53d57ae7c08"
dest_uuid: "fdfeac4190e045f290e003b2298d988d"
mode: GRACEFUL
new_leader_uuid: "ee0d97247d394a8facfa774f15f3a67d"
from {username='slave'} at 127.0.0.1:46074
I20240501 20:01:53.512431 29732 raft_consensus.cc:595] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Received request to transfer leadership to TS ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:53.516238 29732 raft_consensus.cc:1261] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 7 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240501 20:01:53.518894 31448 consensus_queue.cc:1035] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:01:53.521051 28519 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240501 20:01:53.525138 28519 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:01:53.525768 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:01:53.531527 31639 raft_consensus.cc:979] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d: : Instructing follower 4c8da1d580614449a636b855bf11b95f to start an election
I20240501 20:01:53.531940 31639 raft_consensus.cc:1067] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [term 4 LEADER]: Signalling peer 4c8da1d580614449a636b855bf11b95f to start an election
I20240501 20:01:53.534262 29651 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b337b8e3e38d4ad6b1076fbd0d0a1609"
dest_uuid: "4c8da1d580614449a636b855bf11b95f"
from {username='slave'} at 127.0.0.1:59136
I20240501 20:01:53.536381 29651 raft_consensus.cc:482] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:01:53.536770 29651 raft_consensus.cc:3037] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:53.549782 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77be7101c0fe4085b6706296f609870a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:53.550366 29659 raft_consensus.cc:3037] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:53.558580 29659 raft_consensus.cc:2454] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 6.
I20240501 20:01:53.558745 29732 raft_consensus.cc:1261] T a5525fc4b82a44b0991c3dd6098fde81 P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240501 20:01:53.567421 31426 consensus_queue.cc:1035] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:01:53.568598 29772 leader_election.cc:302] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:53.569803 31639 raft_consensus.cc:2781] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Leader election won for term 6
I20240501 20:01:53.572976 31639 raft_consensus.cc:686] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 6 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:53.574470 31639 consensus_queue.cc:237] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:53.571843 31307 leader_election.cc:288] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 6 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:53.580469 29801 raft_consensus.cc:1261] T a5525fc4b82a44b0991c3dd6098fde81 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240501 20:01:53.581735 31426 consensus_queue.cc:1035] T a5525fc4b82a44b0991c3dd6098fde81 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:01:53.566274 31448 raft_consensus.cc:2932] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [term 7 LEADER]: Committing config change with OpId 7.5: config changed from index -1 to 5, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
I20240501 20:01:53.587903 31678 raft_consensus.cc:979] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d: : Instructing follower ee0d97247d394a8facfa774f15f3a67d to start an election
I20240501 20:01:53.588704 31678 raft_consensus.cc:1067] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Signalling peer ee0d97247d394a8facfa774f15f3a67d to start an election
I20240501 20:01:53.590008 29800 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d1d2cac25f354a97b1bfb53d57ae7c08"
dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
from {username='slave'} at 127.0.0.1:41864
I20240501 20:01:53.590550 29800 raft_consensus.cc:482] T d1d2cac25f354a97b1bfb53d57ae7c08 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:01:53.590935 29800 raft_consensus.cc:3037] T d1d2cac25f354a97b1bfb53d57ae7c08 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:53.620348 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77be7101c0fe4085b6706296f609870a" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:53.599503 29800 raft_consensus.cc:504] T d1d2cac25f354a97b1bfb53d57ae7c08 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:53.623515 29800 leader_election.cc:288] T d1d2cac25f354a97b1bfb53d57ae7c08 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:53.643151 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d2cac25f354a97b1bfb53d57ae7c08" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:53.643926 29652 raft_consensus.cc:3037] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:53.661083 29652 raft_consensus.cc:2454] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 5.
I20240501 20:01:53.670327 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1d2cac25f354a97b1bfb53d57ae7c08" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:53.670962 29733 raft_consensus.cc:3032] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Stepping down as leader of term 4
I20240501 20:01:53.671257 29733 raft_consensus.cc:729] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:53.671892 29733 consensus_queue.cc:260] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:53.673481 29733 raft_consensus.cc:3037] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:53.669282 31678 raft_consensus.cc:482] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4c8da1d580614449a636b855bf11b95f)
I20240501 20:01:53.674212 31678 raft_consensus.cc:504] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:53.680634 29733 raft_consensus.cc:2454] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 5.
I20240501 20:01:53.690443 29772 leader_election.cc:302] T d1d2cac25f354a97b1bfb53d57ae7c08 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:53.698056 29554 catalog_manager.cc:5484] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 4 to 6, leader changed from 4c8da1d580614449a636b855bf11b95f (127.27.217.193) to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 6 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:53.635272 29651 raft_consensus.cc:504] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:53.708777 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b337b8e3e38d4ad6b1076fbd0d0a1609" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:53.709367 29733 raft_consensus.cc:3037] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:53.717157 29801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77be7101c0fe4085b6706296f609870a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:53.709648 31445 raft_consensus.cc:2781] T d1d2cac25f354a97b1bfb53d57ae7c08 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:01:53.721952 29651 leader_election.cc:288] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:53.722554 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 45576.6us
I20240501 20:01:53.733844 29733 raft_consensus.cc:2454] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 5.
I20240501 20:01:53.716367 31678 leader_election.cc:288] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
W20240501 20:01:53.736625 29772 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 45549.1us
I20240501 20:01:53.737881 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b337b8e3e38d4ad6b1076fbd0d0a1609" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:53.738605 29800 raft_consensus.cc:3032] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [term 4 LEADER]: Stepping down as leader of term 4
I20240501 20:01:53.738950 29800 raft_consensus.cc:729] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
W20240501 20:01:53.734424 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 62446.1us
I20240501 20:01:53.739686 29800 consensus_queue.cc:260] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:53.730175 31445 raft_consensus.cc:686] T d1d2cac25f354a97b1bfb53d57ae7c08 P ee0d97247d394a8facfa774f15f3a67d [term 5 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:53.742617 31445 consensus_queue.cc:237] T d1d2cac25f354a97b1bfb53d57ae7c08 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:53.743237 29800 raft_consensus.cc:3037] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:53.729305 31268 raft_consensus.cc:482] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:53.760233 31501 raft_consensus.cc:979] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d: : Instructing follower fdfeac4190e045f290e003b2298d988d to start an election
I20240501 20:01:53.760751 31501 raft_consensus.cc:1067] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [term 4 LEADER]: Signalling peer fdfeac4190e045f290e003b2298d988d to start an election
I20240501 20:01:53.756467 31268 raft_consensus.cc:504] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:53.769338 29801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac2171683884f6f97cdfeb0a656cb49" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d" is_pre_election: true
I20240501 20:01:53.770313 29801 raft_consensus.cc:2454] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 5.
I20240501 20:01:53.773205 29651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77be7101c0fe4085b6706296f609870a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4c8da1d580614449a636b855bf11b95f" is_pre_election: true
I20240501 20:01:53.765358 31268 leader_election.cc:288] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:53.774168 29625 leader_election.cc:302] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:53.776389 31426 raft_consensus.cc:2781] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240501 20:01:53.776837 31426 raft_consensus.cc:482] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:01:53.777220 31426 raft_consensus.cc:3037] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:01:53.783689 29809 raft_consensus.cc:2932] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 7 FOLLOWER]: Committing config change with OpId 7.5: config changed from index -1 to 5, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
I20240501 20:01:53.775101 29651 raft_consensus.cc:2354] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d for earlier term 5. Current term is 6.
I20240501 20:01:53.795293 29800 raft_consensus.cc:2454] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 5.
I20240501 20:01:53.795914 29728 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "24a2bba62f434d1c8489b55c8fec2187"
dest_uuid: "fdfeac4190e045f290e003b2298d988d"
from {username='slave'} at 127.0.0.1:36260
I20240501 20:01:53.796559 29728 raft_consensus.cc:482] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:01:53.796977 29728 raft_consensus.cc:3037] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:53.798241 29625 leader_election.cc:302] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:53.799129 29698 leader_election.cc:398] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 5 pre-election: Vote denied by peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959) with higher term. Message: Invalid argument: T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d for earlier term 5. Current term is 6.
I20240501 20:01:53.799351 29698 leader_election.cc:401] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20240501 20:01:53.803490 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac2171683884f6f97cdfeb0a656cb49" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d" is_pre_election: true
I20240501 20:01:53.804311 29732 raft_consensus.cc:2454] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 6.
I20240501 20:01:53.810031 29728 raft_consensus.cc:504] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:53.830328 29728 leader_election.cc:288] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 5 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:53.817065 31447 raft_consensus.cc:2781] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:01:53.846487 31772 raft_consensus.cc:3037] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Advancing to term 6
I20240501 20:01:53.848107 31447 raft_consensus.cc:686] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [term 5 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:53.849035 31447 consensus_queue.cc:237] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:53.892621 31686 raft_consensus.cc:979] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d: : Instructing follower ee0d97247d394a8facfa774f15f3a67d to start an election
I20240501 20:01:53.893118 31686 raft_consensus.cc:1067] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [term 2 LEADER]: Signalling peer ee0d97247d394a8facfa774f15f3a67d to start an election
W20240501 20:01:53.913431 29698 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.193:33959, user_credentials={real_user=slave}} blocked reactor thread for 114959us
I20240501 20:01:53.875867 31426 raft_consensus.cc:504] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:53.918102 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 110253us
I20240501 20:01:53.935767 29801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a2bba62f434d1c8489b55c8fec2187" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:53.936367 29801 raft_consensus.cc:3032] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [term 4 LEADER]: Stepping down as leader of term 4
I20240501 20:01:53.936610 29801 raft_consensus.cc:729] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:53.937168 29801 consensus_queue.cc:260] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:53.938390 29801 raft_consensus.cc:3037] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:53.941237 29652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24a2bba62f434d1c8489b55c8fec2187" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:53.941785 29652 raft_consensus.cc:3037] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:53.948998 29652 raft_consensus.cc:2454] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 5.
I20240501 20:01:53.954078 29805 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aa8cd40cb4044d18a86001ce6ddf97ab"
dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
from {username='slave'} at 127.0.0.1:41864
I20240501 20:01:53.954694 29805 raft_consensus.cc:482] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:01:53.955024 29805 raft_consensus.cc:3037] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:53.967222 29805 raft_consensus.cc:504] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:53.969847 29651 consensus_queue.cc:237] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:53.991686 29801 raft_consensus.cc:2454] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 5.
I20240501 20:01:53.993243 29805 leader_election.cc:288] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:54.000131 29700 leader_election.cc:302] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
W20240501 20:01:54.020443 29625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 64893.8us
W20240501 20:01:54.035555 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 77373.3us
I20240501 20:01:54.042025 31772 raft_consensus.cc:2726] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959) with higher term. Message: Invalid argument: T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfeac4190e045f290e003b2298d988d for earlier term 5. Current term is 6.
I20240501 20:01:54.044618 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac2171683884f6f97cdfeb0a656cb49" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:54.045228 29809 raft_consensus.cc:3037] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Advancing to term 7
I20240501 20:01:54.039150 31678 raft_consensus.cc:2781] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:01:54.048557 31426 leader_election.cc:288] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 7 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:54.063655 29809 raft_consensus.cc:2454] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 7.
I20240501 20:01:54.096408 31678 raft_consensus.cc:686] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [term 5 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:54.097618 29625 leader_election.cc:302] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:54.101881 31789 raft_consensus.cc:2781] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [term 7 FOLLOWER]: Leader election won for term 7
I20240501 20:01:54.102432 31789 raft_consensus.cc:686] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [term 7 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:54.116109 31307 raft_consensus.cc:979] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d: : Instructing follower 4c8da1d580614449a636b855bf11b95f to start an election
I20240501 20:01:54.116601 31307 raft_consensus.cc:1067] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 7 LEADER]: Signalling peer 4c8da1d580614449a636b855bf11b95f to start an election
I20240501 20:01:54.118780 31678 consensus_queue.cc:237] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:54.104980 31789 consensus_queue.cc:237] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:54.186892 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8cd40cb4044d18a86001ce6ddf97ab" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:54.193284 29729 raft_consensus.cc:3032] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:01:54.193567 29729 raft_consensus.cc:729] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:54.194140 29729 consensus_queue.cc:260] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:54.195202 29729 raft_consensus.cc:3037] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:54.203823 29801 raft_consensus.cc:1261] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240501 20:01:54.214615 29729 raft_consensus.cc:2454] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 3.
I20240501 20:01:54.225458 31779 consensus_queue.cc:1035] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20240501 20:01:54.243781 31807 ts_tablet_manager.cc:902] T fc312e7affec4f67ada04db849694bdc P 888d99522e4844118f8f1cf26f268c8c: Initiating tablet copy from peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959)
I20240501 20:01:54.246187 29727 raft_consensus.cc:1261] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240501 20:01:54.270964 29773 leader_election.cc:302] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:54.288218 31307 raft_consensus.cc:2781] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:01:54.289849 31307 raft_consensus.cc:686] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d [term 3 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:54.290648 31307 consensus_queue.cc:237] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:54.299288 31786 consensus_queue.cc:1035] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:01:54.301074 29728 raft_consensus.cc:1261] T d1d2cac25f354a97b1bfb53d57ae7c08 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240501 20:01:54.306653 31773 consensus_queue.cc:1035] T d1d2cac25f354a97b1bfb53d57ae7c08 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
W20240501 20:01:54.317243 31808 log.cc:924] Time spent T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.074s user 0.002s sys 0.000s
I20240501 20:01:54.325287 29557 catalog_manager.cc:5484] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 4 to 5, leader changed from ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 5 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:54.328858 29554 catalog_manager.cc:5484] T d1d2cac25f354a97b1bfb53d57ae7c08 P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 4 to 5, leader changed from fdfeac4190e045f290e003b2298d988d (127.27.217.194) to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 5 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:54.330173 31776 raft_consensus.cc:2932] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [term 6 LEADER]: Committing config change with OpId 6.5: config changed from index -1 to 5, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
I20240501 20:01:54.330550 29554 catalog_manager.cc:5484] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d reported cstate change: config changed from index -1 to 5, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New cstate: current_term: 7 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
I20240501 20:01:54.332971 29801 raft_consensus.cc:2932] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Committing config change with OpId 6.5: config changed from index -1 to 5, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
W20240501 20:01:54.333998 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 297822us
I20240501 20:01:54.357199 29651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8cd40cb4044d18a86001ce6ddf97ab" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:54.357643 29651 raft_consensus.cc:3037] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:01:54.358996 29733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bac2171683884f6f97cdfeb0a656cb49" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:54.359433 29732 raft_consensus.cc:1261] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240501 20:01:54.359607 29733 raft_consensus.cc:3037] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:01:54.371011 29651 raft_consensus.cc:2454] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 3.
I20240501 20:01:54.379029 29729 raft_consensus.cc:2932] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 7 FOLLOWER]: Committing config change with OpId 7.5: config changed from index -1 to 5, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
I20240501 20:01:54.383502 29733 raft_consensus.cc:2454] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 7.
W20240501 20:01:54.392822 31816 log.cc:924] Time spent T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d: Append to log took a long time: real 0.068s user 0.001s sys 0.000s
I20240501 20:01:54.405833 29659 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e49dd5180f04313ab33942fb3ddc147"
dest_uuid: "4c8da1d580614449a636b855bf11b95f"
from {username='slave'} at 127.0.0.1:59136
I20240501 20:01:54.406261 29659 raft_consensus.cc:482] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:01:54.406508 29659 raft_consensus.cc:3037] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [term 7 FOLLOWER]: Advancing to term 8
I20240501 20:01:54.411116 29659 raft_consensus.cc:504] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:54.414135 29659 leader_election.cc:288] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 8 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:54.415495 29805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e49dd5180f04313ab33942fb3ddc147" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: true dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:54.416087 29805 raft_consensus.cc:3032] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 7 LEADER]: Stepping down as leader of term 7
I20240501 20:01:54.416364 29805 raft_consensus.cc:729] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
W20240501 20:01:54.416911 29623 consensus_peers.cc:485] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f -> Peer 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227): Couldn't send request to peer 888d99522e4844118f8f1cf26f268c8c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5525f546426c47a38edf7f56ad04b7ab. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:54.416920 29805 consensus_queue.cc:260] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:54.417479 29623 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.196:37227, user_credentials={real_user=slave}} blocked reactor thread for 94524.8us
I20240501 20:01:54.418031 29805 raft_consensus.cc:3037] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 7 FOLLOWER]: Advancing to term 8
I20240501 20:01:54.423447 29656 raft_consensus.cc:1261] T d1d2cac25f354a97b1bfb53d57ae7c08 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240501 20:01:54.428455 29652 raft_consensus.cc:1261] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240501 20:01:54.450894 29732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e49dd5180f04313ab33942fb3ddc147" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: true dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:54.451709 29732 raft_consensus.cc:3037] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 7 FOLLOWER]: Advancing to term 8
I20240501 20:01:54.459756 29732 raft_consensus.cc:2454] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 8.
I20240501 20:01:54.482051 31786 consensus_queue.cc:1035] T d1d2cac25f354a97b1bfb53d57ae7c08 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:01:54.484609 31501 consensus_queue.cc:1035] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:01:54.487494 29805 raft_consensus.cc:2454] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 8.
I20240501 20:01:54.497321 31812 consensus_queue.cc:1035] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:01:54.503018 29728 consensus_queue.cc:237] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:54.507105 31807 tablet_copy_client.cc:308] T fc312e7affec4f67ada04db849694bdc P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Beginning tablet copy session from remote peer at address 127.27.217.193:33959
I20240501 20:01:54.509047 29670 tablet_copy_service.cc:140] P 4c8da1d580614449a636b855bf11b95f: Received BeginTabletCopySession request for tablet fc312e7affec4f67ada04db849694bdc from peer 888d99522e4844118f8f1cf26f268c8c ({username='slave'} at 127.0.0.1:37936)
I20240501 20:01:54.509533 29670 tablet_copy_service.cc:161] P 4c8da1d580614449a636b855bf11b95f: Beginning new tablet copy session on tablet fc312e7affec4f67ada04db849694bdc from peer 888d99522e4844118f8f1cf26f268c8c at {username='slave'} at 127.0.0.1:37936: session id = 888d99522e4844118f8f1cf26f268c8c-fc312e7affec4f67ada04db849694bdc
I20240501 20:01:54.535014 29670 tablet_copy_source_session.cc:205] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:01:54.540436 31807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc312e7affec4f67ada04db849694bdc. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 20:01:54.544910 29698 consensus_peers.cc:485] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d -> Peer 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227): Couldn't send request to peer 888d99522e4844118f8f1cf26f268c8c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 31fff64b453a423eb0f567b3171564fb. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:54.545435 29801 raft_consensus.cc:1261] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240501 20:01:54.547233 31804 consensus_queue.cc:1035] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:01:54.548826 29556 catalog_manager.cc:5484] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 2 to 3, leader changed from fdfeac4190e045f290e003b2298d988d (127.27.217.194) to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 3 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:54.564736 29656 raft_consensus.cc:1261] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240501 20:01:54.563918 31782 raft_consensus.cc:2932] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [term 5 LEADER]: Committing config change with OpId 5.4: config changed from index -1 to 4, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
W20240501 20:01:54.566349 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 65776.3us
I20240501 20:01:54.566968 29624 leader_election.cc:302] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:54.568279 31807 tablet_copy_client.cc:783] T fc312e7affec4f67ada04db849694bdc P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Starting download of 0 data blocks...
I20240501 20:01:54.555081 29556 catalog_manager.cc:5484] T 5525f546426c47a38edf7f56ad04b7ab P ee0d97247d394a8facfa774f15f3a67d reported cstate change: config changed from index -1 to 5, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New cstate: current_term: 6 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
I20240501 20:01:54.568881 31807 tablet_copy_client.cc:647] T fc312e7affec4f67ada04db849694bdc P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Starting download of 1 WAL segments...
I20240501 20:01:54.574661 31776 raft_consensus.cc:2781] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [term 8 FOLLOWER]: Leader election won for term 8
I20240501 20:01:54.575170 31776 raft_consensus.cc:686] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [term 8 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:54.575958 31776 consensus_queue.cc:237] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:54.584590 31807 tablet_copy_client.cc:523] T fc312e7affec4f67ada04db849694bdc P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:01:54.585580 29801 raft_consensus.cc:2932] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Committing config change with OpId 5.4: config changed from index -1 to 4, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
I20240501 20:01:54.589068 29734 consensus_queue.cc:237] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:54.600241 31782 consensus_queue.cc:1035] T 31fff64b453a423eb0f567b3171564fb P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240501 20:01:54.603062 29729 raft_consensus.cc:1261] T b337b8e3e38d4ad6b1076fbd0d0a1609 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240501 20:01:54.606030 31807 tablet_bootstrap.cc:492] T fc312e7affec4f67ada04db849694bdc P 888d99522e4844118f8f1cf26f268c8c: Bootstrap starting.
W20240501 20:01:54.612438 29698 consensus_peers.cc:485] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d -> Peer 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227): Couldn't send request to peer 888d99522e4844118f8f1cf26f268c8c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 48630ae6eb394ac1b54955dd01cf0ed2. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:54.613389 29801 raft_consensus.cc:1261] T 48630ae6eb394ac1b54955dd01cf0ed2 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240501 20:01:54.616747 31798 consensus_queue.cc:1035] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:01:54.632855 29658 raft_consensus.cc:1261] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240501 20:01:54.642314 29554 catalog_manager.cc:5484] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 4 to 5, leader changed from ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 5 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:54.645411 29554 catalog_manager.cc:5484] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:54.646782 29557 catalog_manager.cc:5484] T 31fff64b453a423eb0f567b3171564fb P ee0d97247d394a8facfa774f15f3a67d reported cstate change: config changed from index -1 to 4, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New cstate: current_term: 5 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
I20240501 20:01:54.660769 31771 consensus_queue.cc:1035] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:01:54.664721 29800 raft_consensus.cc:1261] T bac2171683884f6f97cdfeb0a656cb49 P ee0d97247d394a8facfa774f15f3a67d [term 7 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240501 20:01:54.641335 29728 raft_consensus.cc:2932] T 5525f546426c47a38edf7f56ad04b7ab P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Committing config change with OpId 6.5: config changed from index -1 to 5, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
I20240501 20:01:54.698927 31775 consensus_queue.cc:1035] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.013s
I20240501 20:01:54.684854 31783 consensus_queue.cc:1035] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.025s
I20240501 20:01:54.706905 29800 raft_consensus.cc:1261] T 24a2bba62f434d1c8489b55c8fec2187 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240501 20:01:54.710127 31785 consensus_queue.cc:1035] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:01:54.709666 29805 raft_consensus.cc:2932] T 48630ae6eb394ac1b54955dd01cf0ed2 P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
I20240501 20:01:54.707440 31794 raft_consensus.cc:2932] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
I20240501 20:01:54.695477 29809 raft_consensus.cc:1261] T b337b8e3e38d4ad6b1076fbd0d0a1609 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240501 20:01:54.721372 29728 raft_consensus.cc:1261] T bac2171683884f6f97cdfeb0a656cb49 P fdfeac4190e045f290e003b2298d988d [term 7 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240501 20:01:54.727414 29659 raft_consensus.cc:1261] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 20:01:54.738282 29651 raft_consensus.cc:1261] T 24a2bba62f434d1c8489b55c8fec2187 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240501 20:01:54.755519 31786 consensus_queue.cc:1035] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:01:54.755890 31833 ts_tablet_manager.cc:902] T 5525f546426c47a38edf7f56ad04b7ab P 888d99522e4844118f8f1cf26f268c8c: Initiating tablet copy from peer 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959)
I20240501 20:01:54.746963 31792 consensus_queue.cc:1035] T b337b8e3e38d4ad6b1076fbd0d0a1609 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.069s
I20240501 20:01:54.754411 31775 consensus_queue.cc:1035] T bac2171683884f6f97cdfeb0a656cb49 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.058s
I20240501 20:01:54.751333 31804 consensus_queue.cc:1035] T 24a2bba62f434d1c8489b55c8fec2187 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.050s
I20240501 20:01:54.917045 29733 raft_consensus.cc:1261] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240501 20:01:54.924868 31470 consensus_queue.cc:1035] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 20:01:55.006028 29658 raft_consensus.cc:2932] T 31fff64b453a423eb0f567b3171564fb P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Committing config change with OpId 5.4: config changed from index -1 to 4, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } attrs { replace: true } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
I20240501 20:01:55.021312 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:01:55.137617 29554 catalog_manager.cc:5484] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 7 to 8, leader changed from ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 8 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:55.158002 29729 raft_consensus.cc:1261] T 4e49dd5180f04313ab33942fb3ddc147 P fdfeac4190e045f290e003b2298d988d [term 8 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240501 20:01:55.158195 29809 raft_consensus.cc:1261] T 4e49dd5180f04313ab33942fb3ddc147 P ee0d97247d394a8facfa774f15f3a67d [term 8 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240501 20:01:55.168941 31848 ts_tablet_manager.cc:902] T 31fff64b453a423eb0f567b3171564fb P 888d99522e4844118f8f1cf26f268c8c: Initiating tablet copy from peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:55.162698 31776 consensus_queue.cc:1035] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
F20240501 20:01:55.169821 31848 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (init.queue_time_us, init.queue_time_us) (init.run_cpu_time_us, init.run_cpu_time_us) (init.run_wall_time_us, init.run_wall_time_us) (leader-initialization.queue_time_us, leader-initialization.queue_time_us) (leader-initialization.run_cpu_time_us, leader-initialization.run_cpu_time_us) (leader-initialization.run_wall_time_us, leader-initialization.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-25e07220db7444ae97f294d70eda6a93.queue_time_us, tablet-download-pool-25e07220db7444ae97f294d70eda6a93.queue_time_us) (tablet-download-pool-25e07220db7444ae97f294d70eda6a93.run_cpu_time_us, tablet-download-pool-25e07220db7444ae97f294d70eda6a93.run_cpu_time_us) (tablet-download-pool-25e07220db7444ae97f294d70eda6a93.run_wall_time_us, tablet-download-pool-25e07220db7444ae97f294d70eda6a93.run_wall_time_us) (tablet-download-pool-31fff64b453a423eb0f567b3171564fb.queue_time_us, tablet-download-pool-31fff64b453a423eb0f567b3171564fb.queue_time_us) (tablet-download-pool-5525f546426c47a38edf7f56ad04b7ab.queue_time_us, tablet-download-pool-5525f546426c47a38edf7f56ad04b7ab.queue_time_us) (tablet-download-pool-5525f546426c47a38edf7f56ad04b7ab.run_cpu_time_us, tablet-download-pool-5525f546426c47a38edf7f56ad04b7ab.run_cpu_time_us) (tablet-download-pool-5525f546426c47a38edf7f56ad04b7ab.run_wall_time_us, tablet-download-pool-5525f546426c47a38edf7f56ad04b7ab.run_wall_time_us) (tablet-download-pool-696121417ea9403990456cfc6fc4f416.queue_time_us, tablet-download-pool-696121417ea9403990456cfc6fc4f416.queue_time_us) (tablet-download-pool-696121417ea9403990456cfc6fc4f416.run_cpu_time_us, tablet-download-pool-696121417ea9403990456cfc6fc4f416.run_cpu_time_us) (tablet-download-pool-696121417ea9403990456cfc6fc4f416.run_wall_time_us, tablet-download-pool-696121417ea9403990456cfc6fc4f416.run_wall_time_us) (tablet-download-pool-77be7101c0fe4085b6706296f609870a.queue_time_us, tablet-download-pool-77be7101c0fe4085b6706296f609870a.queue_time_us) (tablet-download-pool-77be7101c0fe4085b6706296f609870a.run_cpu_time_us, tablet-download-pool-77be7101c0fe4085b6706296f609870a.run_cpu_time_us) (tablet-download-pool-77be7101c0fe4085b6706296f609870a.run_wall_time_us, tablet-download-pool-77be7101c0fe4085b6706296f609870a.run_wall_time_us) (tablet-download-pool-7ed27033ab1645e9ab0600ab635d9589.queue_time_us, tablet-download-pool-7ed27033ab1645e9ab0600ab635d9589.queue_time_us) (tablet-download-pool-7ed27033ab1645e9ab0600ab635d9589.run_cpu_time_us, tablet-download-pool-7ed27033ab1645e9ab0600ab635d9589.run_cpu_time_us) (tablet-download-pool-7ed27033ab1645e9ab0600ab635d9589.run_wall_time_us, tablet-download-pool-7ed27033ab1645e9ab0600ab635d9589.run_wall_time_us) (tablet-download-pool-8bd5e83faaff4e2da4e4cc22ba410d33.queue_time_us, tablet-download-pool-8bd5e83faaff4e2da4e4cc22ba410d33.queue_time_us) (tablet-download-pool-8bd5e83faaff4e2da4e4cc22ba410d33.run_cpu_time_us, tablet-download-pool-8bd5e83faaff4e2da4e4cc22ba410d33.run_cpu_time_us) (tablet-download-pool-8bd5e83faaff4e2da4e4cc22ba410d33.run_wall_time_us, tablet-download-pool-8bd5e83faaff4e2da4e4cc22ba410d33.run_wall_time_us) (tablet-download-pool-9046e38fc57f4364aa3d539e1fc01928.queue_time_us, tablet-download-pool-9046e38fc57f4364aa3d539e1fc01928.queue_time_us) (tablet-download-pool-9046e38fc57f4364aa3d539e1fc01928.run_cpu_time_us, tablet-download-pool-9046e38fc57f4364aa3d539e1fc01928.run_cpu_time_us) (tablet-download-pool-9046e38fc57f4364aa3d539e1fc01928.run_wall_time_us, tablet-download-pool-9046e38fc57f4364aa3d539e1fc01928.run_wall_time_us) (tablet-download-pool-add9d7579351483fb268f89a44e95d8a.queue_time_us, tablet-download-pool-add9d7579351483fb268f89a44e95d8a.queue_time_us) (tablet-download-pool-add9d7579351483fb268f89a44e95d8a.run_cpu_time_us, tablet-download-pool-add9d7579351483fb268f89a44e95d8a.run_cpu_time_us) (tablet-download-pool-add9d7579351483fb268f89a44e95d8a.run_wall_time_us, tablet-download-pool-add9d7579351483fb268f89a44e95d8a.run_wall_time_us) (tablet-download-pool-be383be783234855878cb8fd442c7b05.queue_time_us, tablet-download-pool-be383be783234855878cb8fd442c7b05.queue_time_us) (tablet-download-pool-be383be783234855878cb8fd442c7b05.run_cpu_time_us, tablet-download-pool-be383be783234855878cb8fd442c7b05.run_cpu_time_us) (tablet-download-pool-be383be783234855878cb8fd442c7b05.run_wall_time_us, tablet-download-pool-be383be783234855878cb8fd442c7b05.run_wall_time_us) (tablet-download-pool-cb05df43bdfa479b902b1da297fb5974.queue_time_us, tablet-download-pool-cb05df43bdfa479b902b1da297fb5974.queue_time_us) (tablet-download-pool-cb05df43bdfa479b902b1da297fb5974.run_cpu_time_us, tablet-download-pool-cb05df43bdfa479b902b1da297fb5974.run_cpu_time_us) (tablet-download-pool-cb05df43bdfa479b902b1da297fb5974.run_wall_time_us, tablet-download-pool-cb05df43bdfa479b902b1da297fb5974.run_wall_time_us) (tablet-download-pool-fa7e89c0572e4ee7be8e5315b459bdb3.queue_time_us, tablet-download-pool-fa7e89c0572e4ee7be8e5315b459bdb3.queue_time_us) (tablet-download-pool-fa7e89c0572e4ee7be8e5315b459bdb3.run_cpu_time_us, tablet-download-pool-fa7e89c0572e4ee7be8e5315b459bdb3.run_cpu_time_us) (tablet-download-pool-fa7e89c0572e4ee7be8e5315b459bdb3.run_wall_time_us, tablet-download-pool-fa7e89c0572e4ee7be8e5315b459bdb3.run_wall_time_us) (tablet-download-pool-fc312e7affec4f67ada04db849694bdc.queue_time_us, tablet-download-pool-fc312e7affec4f67ada04db849694bdc.queue_time_us) (tablet-download-pool-fc312e7affec4f67ada04db849694bdc.run_cpu_time_us, tablet-download-pool-fc312e7affec4f67ada04db849694bdc.run_cpu_time_us) (tablet-download-pool-fc312e7affec4f67ada04db849694bdc.run_wall_time_us, tablet-download-pool-fc312e7affec4f67ada04db849694bdc.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
*** Aborted at 1714593715 (unix time) try "date -d @1714593715" if you are using GNU date ***
I20240501 20:01:55.171500 31783 consensus_queue.cc:1035] T 4e49dd5180f04313ab33942fb3ddc147 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:01:55.162155 31833 tablet_copy_client.cc:308] T 5525f546426c47a38edf7f56ad04b7ab P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Beginning tablet copy session from remote peer at address 127.27.217.193:33959
I20240501 20:01:55.257870 29670 tablet_copy_service.cc:140] P 4c8da1d580614449a636b855bf11b95f: Received BeginTabletCopySession request for tablet 5525f546426c47a38edf7f56ad04b7ab from peer 888d99522e4844118f8f1cf26f268c8c ({username='slave'} at 127.0.0.1:37936)
I20240501 20:01:55.258422 29670 tablet_copy_service.cc:161] P 4c8da1d580614449a636b855bf11b95f: Beginning new tablet copy session on tablet 5525f546426c47a38edf7f56ad04b7ab from peer 888d99522e4844118f8f1cf26f268c8c at {username='slave'} at 127.0.0.1:37936: session id = 888d99522e4844118f8f1cf26f268c8c-5525f546426c47a38edf7f56ad04b7ab
I20240501 20:01:55.224714 29660 raft_consensus.cc:2932] T 48630ae6eb394ac1b54955dd01cf0ed2 P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } } }
I20240501 20:01:55.282243 29670 tablet_copy_source_session.cc:205] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:01:55.288846 31833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5525f546426c47a38edf7f56ad04b7ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:55.311388 29557 catalog_manager.cc:5484] T 48630ae6eb394ac1b54955dd01cf0ed2 P fdfeac4190e045f290e003b2298d988d reported cstate change: config changed from index -1 to 4, NON_VOTER 888d99522e4844118f8f1cf26f268c8c (127.27.217.196) added. New cstate: current_term: 3 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:55.329320 31858 ts_tablet_manager.cc:902] T 48630ae6eb394ac1b54955dd01cf0ed2 P 888d99522e4844118f8f1cf26f268c8c: Initiating tablet copy from peer fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:55.336872 31833 tablet_copy_client.cc:783] T 5525f546426c47a38edf7f56ad04b7ab P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Starting download of 0 data blocks...
I20240501 20:01:55.337493 31833 tablet_copy_client.cc:647] T 5525f546426c47a38edf7f56ad04b7ab P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Starting download of 1 WAL segments...
I20240501 20:01:55.355808 31833 tablet_copy_client.cc:523] T 5525f546426c47a38edf7f56ad04b7ab P 888d99522e4844118f8f1cf26f268c8c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:01:55.369462 31833 tablet_bootstrap.cc:492] T 5525f546426c47a38edf7f56ad04b7ab P 888d99522e4844118f8f1cf26f268c8c: Bootstrap starting.
W20240501 20:01:55.435914 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 50375.1us
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800006f67) received by PID 28519 (TID 0x7fe36f967700) from PID 28519; stack trace: ***
@ 0x2b0110 __tsan::CallUserSignalHandler() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
@ 0x2b25f3 rtl_sigaction() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
I20240501 20:01:55.772241 29800 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0374bb790ed345ed90452595f71383a2"
dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
mode: GRACEFUL
new_leader_uuid: "4c8da1d580614449a636b855bf11b95f"
from {username='slave'} at 127.0.0.1:33582
I20240501 20:01:55.773212 29800 raft_consensus.cc:595] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 6 LEADER]: Received request to transfer leadership to TS 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:55.776324 29800 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22"
dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
mode: GRACEFUL
new_leader_uuid: "4c8da1d580614449a636b855bf11b95f"
from {username='slave'} at 127.0.0.1:33582
I20240501 20:01:55.776867 29800 raft_consensus.cc:595] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 7 LEADER]: Received request to transfer leadership to TS 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:55.780112 29800 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9472c53f79ec423f82c396bd980674e2"
dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
mode: GRACEFUL
new_leader_uuid: "fdfeac4190e045f290e003b2298d988d"
from {username='slave'} at 127.0.0.1:33582
I20240501 20:01:55.780622 29800 raft_consensus.cc:595] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [term 5 LEADER]: Received request to transfer leadership to TS fdfeac4190e045f290e003b2298d988d
I20240501 20:01:55.786476 29732 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a7c597f15e844f30837012baed4b0501"
dest_uuid: "fdfeac4190e045f290e003b2298d988d"
mode: GRACEFUL
new_leader_uuid: "ee0d97247d394a8facfa774f15f3a67d"
from {username='slave'} at 127.0.0.1:46074
I20240501 20:01:55.787012 29732 raft_consensus.cc:595] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [term 6 LEADER]: Received request to transfer leadership to TS ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:55.796765 29658 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc"
dest_uuid: "4c8da1d580614449a636b855bf11b95f"
mode: GRACEFUL
new_leader_uuid: "fdfeac4190e045f290e003b2298d988d"
from {username='slave'} at 127.0.0.1:60736
I20240501 20:01:55.797302 29658 raft_consensus.cc:595] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [term 7 LEADER]: Received request to transfer leadership to TS fdfeac4190e045f290e003b2298d988d
I20240501 20:01:55.805215 28519 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240501 20:01:55.822023 28519 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:01:55.822618 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
@ 0x7fe457c81980 (unknown) at ??:0
I20240501 20:01:55.846864 31796 raft_consensus.cc:979] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d: : Instructing follower 4c8da1d580614449a636b855bf11b95f to start an election
I20240501 20:01:55.847543 31791 raft_consensus.cc:1067] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 7 LEADER]: Signalling peer 4c8da1d580614449a636b855bf11b95f to start an election
I20240501 20:01:55.849766 29651 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22"
dest_uuid: "4c8da1d580614449a636b855bf11b95f"
from {username='slave'} at 127.0.0.1:59136
I20240501 20:01:55.850432 29651 raft_consensus.cc:482] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:01:55.851035 29651 raft_consensus.cc:3037] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 7 FOLLOWER]: Advancing to term 8
I20240501 20:01:55.872412 29651 raft_consensus.cc:504] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:55.878132 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: true dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:55.878902 29728 raft_consensus.cc:3037] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 7 FOLLOWER]: Advancing to term 8
I20240501 20:01:55.888166 29728 raft_consensus.cc:2454] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 8.
I20240501 20:01:55.898679 31307 raft_consensus.cc:979] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d: : Instructing follower 4c8da1d580614449a636b855bf11b95f to start an election
I20240501 20:01:55.908545 31307 raft_consensus.cc:1067] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 6 LEADER]: Signalling peer 4c8da1d580614449a636b855bf11b95f to start an election
I20240501 20:01:55.899652 31771 raft_consensus.cc:979] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d: : Instructing follower ee0d97247d394a8facfa774f15f3a67d to start an election
I20240501 20:01:55.901288 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be5f38dc96d4fad80d83eea0822aa22" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: true dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:55.909575 29809 raft_consensus.cc:3032] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 7 LEADER]: Stepping down as leader of term 7
I20240501 20:01:55.909927 29809 raft_consensus.cc:729] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:55.909544 31771 raft_consensus.cc:1067] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [term 6 LEADER]: Signalling peer ee0d97247d394a8facfa774f15f3a67d to start an election
I20240501 20:01:55.909862 29624 leader_election.cc:302] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:55.910779 29809 consensus_queue.cc:260] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:55.911738 29800 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a7c597f15e844f30837012baed4b0501"
dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
from {username='slave'} at 127.0.0.1:41864
I20240501 20:01:55.912370 29800 raft_consensus.cc:482] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:01:55.912434 31775 raft_consensus.cc:2781] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 8 FOLLOWER]: Leader election won for term 8
I20240501 20:01:55.912761 29800 raft_consensus.cc:3037] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:01:55.912978 31775 raft_consensus.cc:686] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [term 8 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:55.913355 29659 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0374bb790ed345ed90452595f71383a2"
dest_uuid: "4c8da1d580614449a636b855bf11b95f"
from {username='slave'} at 127.0.0.1:59136
I20240501 20:01:55.913818 29659 raft_consensus.cc:482] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:01:55.914165 29659 raft_consensus.cc:3037] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:01:55.913899 31775 consensus_queue.cc:237] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:55.911895 29809 raft_consensus.cc:3037] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 7 FOLLOWER]: Advancing to term 8
I20240501 20:01:55.901948 29651 leader_election.cc:288] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 8 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:55.926291 31860 raft_consensus.cc:979] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f: : Instructing follower fdfeac4190e045f290e003b2298d988d to start an election
I20240501 20:01:55.926774 31860 raft_consensus.cc:1067] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [term 7 LEADER]: Signalling peer fdfeac4190e045f290e003b2298d988d to start an election
I20240501 20:01:55.930542 29728 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc"
dest_uuid: "fdfeac4190e045f290e003b2298d988d"
from {username='slave'} at 127.0.0.1:36270
I20240501 20:01:55.932883 29728 raft_consensus.cc:482] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:01:55.933259 29728 raft_consensus.cc:3037] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 7 FOLLOWER]: Advancing to term 8
@ 0x7fe4511f5fb7 gsignal at ??:0
I20240501 20:01:55.938135 29800 raft_consensus.cc:504] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:55.940937 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c597f15e844f30837012baed4b0501" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: true dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:55.941493 29660 raft_consensus.cc:3037] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:01:55.941309 29659 raft_consensus.cc:504] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:55.943656 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0374bb790ed345ed90452595f71383a2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:55.944124 29729 raft_consensus.cc:3037] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:01:55.961723 29728 raft_consensus.cc:504] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:55.963902 29659 leader_election.cc:288] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 7 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:55.964253 29807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0374bb790ed345ed90452595f71383a2" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:55.964898 29807 raft_consensus.cc:3032] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 6 LEADER]: Stepping down as leader of term 6
I20240501 20:01:55.965183 29807 raft_consensus.cc:729] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:55.966172 29729 raft_consensus.cc:2454] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 7.
I20240501 20:01:55.967476 29807 consensus_queue.cc:260] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:55.968626 29807 raft_consensus.cc:3037] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:01:55.956835 29809 raft_consensus.cc:2454] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 8.
I20240501 20:01:55.972680 29624 leader_election.cc:302] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:55.973589 31862 raft_consensus.cc:2781] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 7 FOLLOWER]: Leader election won for term 7
I20240501 20:01:55.973994 31862 raft_consensus.cc:686] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [term 7 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:55.974825 31862 consensus_queue.cc:237] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:55.976384 29729 raft_consensus.cc:1226] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [term 7 FOLLOWER]: Rejecting Update request from peer ee0d97247d394a8facfa774f15f3a67d for earlier term 6. Current term is 7. Ops: []
I20240501 20:01:55.976752 29660 raft_consensus.cc:2454] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 7.
I20240501 20:01:55.978653 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7c597f15e844f30837012baed4b0501" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: true dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:55.979254 29727 raft_consensus.cc:3032] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [term 6 LEADER]: Stepping down as leader of term 6
I20240501 20:01:55.979543 29727 raft_consensus.cc:729] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:55.980316 29727 consensus_queue.cc:260] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:55.981410 29727 raft_consensus.cc:3037] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:01:55.981593 29807 raft_consensus.cc:2454] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 7.
I20240501 20:01:55.985674 29772 leader_election.cc:302] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:55.987489 31796 raft_consensus.cc:2781] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [term 7 FOLLOWER]: Leader election won for term 7
I20240501 20:01:55.988082 31796 raft_consensus.cc:686] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [term 7 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:55.989874 31796 consensus_queue.cc:237] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:55.996237 29800 leader_election.cc:288] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 7 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:55.999246 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: true dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:55.999822 29659 raft_consensus.cc:3032] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [term 7 LEADER]: Stepping down as leader of term 7
I20240501 20:01:56.000136 29659 raft_consensus.cc:729] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:56.001000 29659 consensus_queue.cc:260] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:56.005551 29659 raft_consensus.cc:3037] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [term 7 FOLLOWER]: Advancing to term 8
@ 0x7fe4511f7921 abort at ??:0
@ 0x2b6106 abort at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
I20240501 20:01:56.015985 29727 raft_consensus.cc:2454] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 7.
I20240501 20:01:56.016821 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: true dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:56.017480 29800 raft_consensus.cc:3037] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 7 FOLLOWER]: Advancing to term 8
I20240501 20:01:56.031034 29800 raft_consensus.cc:2454] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 8.
I20240501 20:01:56.027293 29728 leader_election.cc:288] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 8 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:56.039151 29659 raft_consensus.cc:2454] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 8.
I20240501 20:01:56.040503 29698 leader_election.cc:302] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:56.041618 31771 raft_consensus.cc:2781] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 8 FOLLOWER]: Leader election won for term 8
I20240501 20:01:56.042181 31771 raft_consensus.cc:686] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 8 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:56.043148 31771 consensus_queue.cc:237] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:56.074100 31307 raft_consensus.cc:979] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d: : Instructing follower fdfeac4190e045f290e003b2298d988d to start an election
I20240501 20:01:56.074635 31307 raft_consensus.cc:1067] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [term 5 LEADER]: Signalling peer fdfeac4190e045f290e003b2298d988d to start an election
I20240501 20:01:56.078409 29728 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9472c53f79ec423f82c396bd980674e2"
dest_uuid: "fdfeac4190e045f290e003b2298d988d"
from {username='slave'} at 127.0.0.1:36260
I20240501 20:01:56.079032 29728 raft_consensus.cc:482] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:01:56.079408 29728 raft_consensus.cc:3037] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:56.084887 29556 catalog_manager.cc:5484] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 6 to 7, leader changed from fdfeac4190e045f290e003b2298d988d (127.27.217.194) to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 7 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:56.088821 29557 catalog_manager.cc:5484] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 7 to 8, leader changed from ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 8 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:56.096323 29728 raft_consensus.cc:504] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:56.106280 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9472c53f79ec423f82c396bd980674e2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:56.107018 29660 raft_consensus.cc:3037] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:56.108500 29728 leader_election.cc:288] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:56.110893 29800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9472c53f79ec423f82c396bd980674e2" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:56.111517 29800 raft_consensus.cc:3032] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [term 5 LEADER]: Stepping down as leader of term 5
I20240501 20:01:56.111804 29800 raft_consensus.cc:729] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:56.117964 29800 consensus_queue.cc:260] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:56.119444 29800 raft_consensus.cc:3037] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:56.132803 29800 raft_consensus.cc:2454] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 6.
I20240501 20:01:56.133924 29700 leader_election.cc:302] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:56.134830 31794 raft_consensus.cc:2781] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Leader election won for term 6
I20240501 20:01:56.109076 29554 catalog_manager.cc:5484] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 7 to 8, leader changed from 4c8da1d580614449a636b855bf11b95f (127.27.217.193) to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 8 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:56.135803 31794 raft_consensus.cc:686] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [term 6 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:56.151451 31794 consensus_queue.cc:237] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:56.161294 29660 raft_consensus.cc:2454] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 6.
@ 0x7fe453febda4 google::LogMessage::SendToLog() at ??:0
@ 0x7fe453fec950 google::LogMessage::Flush() at ??:0
@ 0x7fe453ff1a8b google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240501 20:01:56.355197 29727 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc"
dest_uuid: "fdfeac4190e045f290e003b2298d988d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36258
I20240501 20:01:56.355785 29727 raft_consensus.cc:595] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 8 LEADER]: Received request to transfer leadership
@ 0x7fe4549793c7 kudu::TraceMetrics::InternName() at ??:0
I20240501 20:01:56.403384 29554 catalog_manager.cc:5484] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 5 to 6, leader changed from ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 6 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:56.425530 29557 catalog_manager.cc:5484] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 6 to 7, leader changed from ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 7 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:56.436921 29729 raft_consensus.cc:1261] T 0374bb790ed345ed90452595f71383a2 P fdfeac4190e045f290e003b2298d988d [term 7 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
@ 0x7fe45494f94f kudu::ThreadPool::ThreadPool() at ??:0
I20240501 20:01:56.443974 31862 consensus_queue.cc:1035] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240501 20:01:56.465512 29809 raft_consensus.cc:1261] T 0374bb790ed345ed90452595f71383a2 P ee0d97247d394a8facfa774f15f3a67d [term 7 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240501 20:01:56.467861 31862 consensus_queue.cc:1035] T 0374bb790ed345ed90452595f71383a2 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
@ 0x7fe45494d0a5 kudu::ThreadPoolBuilder::Build() at ??:0
I20240501 20:01:56.478912 29728 raft_consensus.cc:1261] T 7be5f38dc96d4fad80d83eea0822aa22 P fdfeac4190e045f290e003b2298d988d [term 8 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240501 20:01:56.480559 29800 raft_consensus.cc:1261] T 7be5f38dc96d4fad80d83eea0822aa22 P ee0d97247d394a8facfa774f15f3a67d [term 8 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240501 20:01:56.483685 31862 consensus_queue.cc:1035] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240501 20:01:56.484711 31447 consensus_queue.cc:1035] T 7be5f38dc96d4fad80d83eea0822aa22 P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240501 20:01:56.495880 29728 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc"
dest_uuid: "fdfeac4190e045f290e003b2298d988d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36258
I20240501 20:01:56.496583 29728 raft_consensus.cc:595] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 8 LEADER]: Received request to transfer leadership
W20240501 20:01:56.518289 29543 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for fc312e7affec4f67ada04db849694bdc already in progress [suppressed 9 similar messages]
@ 0x7fe45c76d422 kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240501 20:01:56.572922 29659 raft_consensus.cc:1261] T 9472c53f79ec423f82c396bd980674e2 P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240501 20:01:56.574610 31771 consensus_queue.cc:1035] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:01:56.579941 29807 raft_consensus.cc:1261] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 8 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20240501 20:01:56.581676 31771 consensus_queue.cc:1035] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:01:56.622668 29660 raft_consensus.cc:1261] T a7c597f15e844f30837012baed4b0501 P 4c8da1d580614449a636b855bf11b95f [term 7 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240501 20:01:56.624051 31307 consensus_queue.cc:1035] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.006s
I20240501 20:01:56.650086 29728 raft_consensus.cc:1261] T a7c597f15e844f30837012baed4b0501 P fdfeac4190e045f290e003b2298d988d [term 7 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
@ 0x7fe45c773c89 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240501 20:01:56.684641 31796 consensus_queue.cc:1035] T a7c597f15e844f30837012baed4b0501 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240501 20:01:56.718050 31771 raft_consensus.cc:979] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d: : Instructing follower ee0d97247d394a8facfa774f15f3a67d to start an election
I20240501 20:01:56.718595 31771 raft_consensus.cc:1067] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 8 LEADER]: Signalling peer ee0d97247d394a8facfa774f15f3a67d to start an election
W20240501 20:01:56.745432 29773 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 86313.3us
W20240501 20:01:56.798017 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 59043us
I20240501 20:01:56.800021 29807 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc"
dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
from {username='slave'} at 127.0.0.1:41864
I20240501 20:01:56.800097 29809 raft_consensus.cc:1261] T 9472c53f79ec423f82c396bd980674e2 P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240501 20:01:56.800704 29807 raft_consensus.cc:482] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:01:56.801110 29807 raft_consensus.cc:3037] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 8 FOLLOWER]: Advancing to term 9
I20240501 20:01:56.841629 29807 raft_consensus.cc:504] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:56.855594 29807 leader_election.cc:288] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 9 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:56.863345 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: true dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:56.871016 29727 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc"
dest_uuid: "fdfeac4190e045f290e003b2298d988d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36258
I20240501 20:01:56.922931 29659 raft_consensus.cc:1261] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [term 8 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20240501 20:01:56.931131 31918 consensus_queue.cc:1035] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20240501 20:01:56.943006 29624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 136556us
W20240501 20:01:56.952167 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 136284us
W20240501 20:01:57.003412 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 43813.2us
I20240501 20:01:57.014801 29727 raft_consensus.cc:595] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 8 LEADER]: Received request to transfer leadership
I20240501 20:01:57.015462 29728 raft_consensus.cc:3032] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 8 LEADER]: Stepping down as leader of term 8
I20240501 20:01:57.015908 29728 raft_consensus.cc:729] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:57.016934 29728 consensus_queue.cc:260] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
I20240501 20:01:57.032474 31915 consensus_queue.cc:1035] T 9472c53f79ec423f82c396bd980674e2 P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.008s
W20240501 20:01:57.033816 30384 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.194:32907, user_credentials={real_user=slave}} blocked reactor thread for 132567us
W20240501 20:01:57.041330 29623 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.196:37227, user_credentials={real_user=slave}} blocked reactor thread for 130755us
W20240501 20:01:57.066067 29606 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for fc312e7affec4f67ada04db849694bdc already in progress
W20240501 20:01:57.066434 29606 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for fc312e7affec4f67ada04db849694bdc already in progress
I20240501 20:01:57.070861 29728 raft_consensus.cc:3037] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 8 FOLLOWER]: Advancing to term 9
I20240501 20:01:57.075685 29660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc312e7affec4f67ada04db849694bdc" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: true dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:57.091748 29728 raft_consensus.cc:2454] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 9.
I20240501 20:01:57.106089 29773 leader_election.cc:302] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:57.107013 31791 raft_consensus.cc:2781] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 9 FOLLOWER]: Leader election won for term 9
I20240501 20:01:57.107723 31791 raft_consensus.cc:686] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [term 9 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:57.108939 31791 consensus_queue.cc:237] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } }
@ 0x7fe45c8094ef kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240501 20:01:57.146098 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 20:01:57.214253 29556 catalog_manager.cc:5484] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 8 to 9, leader changed from fdfeac4190e045f290e003b2298d988d (127.27.217.194) to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 9 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "888d99522e4844118f8f1cf26f268c8c" member_type: NON_VOTER last_known_addr { host: "127.27.217.196" port: 37227 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
@ 0x7fe45c83a3cf kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7fe45c8201ce kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
I20240501 20:01:57.365980 29727 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "211e27a3b0084b89a6dd7550849b5a65"
dest_uuid: "fdfeac4190e045f290e003b2298d988d"
mode: GRACEFUL
new_leader_uuid: "ee0d97247d394a8facfa774f15f3a67d"
from {username='slave'} at 127.0.0.1:46074
I20240501 20:01:57.366559 29727 raft_consensus.cc:595] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Received request to transfer leadership to TS ee0d97247d394a8facfa774f15f3a67d
I20240501 20:01:57.369678 29800 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "77be7101c0fe4085b6706296f609870a"
dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
mode: GRACEFUL
new_leader_uuid: "4c8da1d580614449a636b855bf11b95f"
from {username='slave'} at 127.0.0.1:33582
I20240501 20:01:57.370234 29800 raft_consensus.cc:595] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 6 LEADER]: Received request to transfer leadership to TS 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:57.373023 29658 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "84fa28fb55f44b61ba1a88ed775a78ba"
dest_uuid: "4c8da1d580614449a636b855bf11b95f"
mode: GRACEFUL
new_leader_uuid: "fdfeac4190e045f290e003b2298d988d"
from {username='slave'} at 127.0.0.1:60736
I20240501 20:01:57.373636 29658 raft_consensus.cc:595] T 84fa28fb55f44b61ba1a88ed775a78ba P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Received request to transfer leadership to TS fdfeac4190e045f290e003b2298d988d
I20240501 20:01:57.376438 29658 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9f261d2a05bf4c838532049601694b4a"
dest_uuid: "4c8da1d580614449a636b855bf11b95f"
mode: GRACEFUL
new_leader_uuid: "fdfeac4190e045f290e003b2298d988d"
from {username='slave'} at 127.0.0.1:60736
I20240501 20:01:57.376989 29658 raft_consensus.cc:595] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [term 5 LEADER]: Received request to transfer leadership to TS fdfeac4190e045f290e003b2298d988d
I20240501 20:01:57.379380 29800 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aa8cd40cb4044d18a86001ce6ddf97ab"
dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
mode: GRACEFUL
new_leader_uuid: "4c8da1d580614449a636b855bf11b95f"
from {username='slave'} at 127.0.0.1:33582
I20240501 20:01:57.379885 29800 raft_consensus.cc:595] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d [term 3 LEADER]: Received request to transfer leadership to TS 4c8da1d580614449a636b855bf11b95f
I20240501 20:01:57.381209 28519 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240501 20:01:57.383044 28519 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 20:01:57.383710 28519 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
@ 0x7fe45c82017a _ZNSt3__18__invokeIRZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBENS_8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEE4$_12JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSH_DpOSI_ at ??:0
@ 0x7fe45c82010a std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
W20240501 20:01:57.497187 29623 consensus_peers.cc:485] T 5525f546426c47a38edf7f56ad04b7ab P 4c8da1d580614449a636b855bf11b95f -> Peer 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227): Couldn't send request to peer 888d99522e4844118f8f1cf26f268c8c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
@ 0x7fe45c8200d2 std::__1::__function::__alloc_func<>::operator()() at ??:0
@ 0x7fe45c81f2fe std::__1::__function::__func<>::operator()() at ??:0
I20240501 20:01:57.616362 31445 raft_consensus.cc:979] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d: : Instructing follower 4c8da1d580614449a636b855bf11b95f to start an election
I20240501 20:01:57.616853 31904 raft_consensus.cc:1067] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 6 LEADER]: Signalling peer 4c8da1d580614449a636b855bf11b95f to start an election
I20240501 20:01:57.618317 29659 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "77be7101c0fe4085b6706296f609870a"
dest_uuid: "4c8da1d580614449a636b855bf11b95f"
from {username='slave'} at 127.0.0.1:59136
I20240501 20:01:57.618868 29659 raft_consensus.cc:482] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:01:57.619244 29659 raft_consensus.cc:3037] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:01:57.626014 29659 raft_consensus.cc:504] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:57.633682 29729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77be7101c0fe4085b6706296f609870a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:57.634297 29729 raft_consensus.cc:3037] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:01:57.639755 29729 raft_consensus.cc:2454] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 7.
I20240501 20:01:57.641266 29624 leader_election.cc:302] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:57.642055 31924 raft_consensus.cc:2781] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 7 FOLLOWER]: Leader election won for term 7
I20240501 20:01:57.642531 31924 raft_consensus.cc:686] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [term 7 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:57.643255 31924 consensus_queue.cc:237] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:57.649140 29659 leader_election.cc:288] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 7 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:57.656608 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77be7101c0fe4085b6706296f609870a" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:57.657188 29809 raft_consensus.cc:3032] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 6 LEADER]: Stepping down as leader of term 6
I20240501 20:01:57.657406 29809 raft_consensus.cc:729] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:57.660017 29809 consensus_queue.cc:260] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
@ 0x7fe45dbc5da5 std::__1::__function::__value_func<>::operator()() at ??:0
I20240501 20:01:57.661296 29809 raft_consensus.cc:3037] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:01:57.667974 29809 raft_consensus.cc:2454] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 7.
I20240501 20:01:57.682662 29556 catalog_manager.cc:5484] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 6 to 7, leader changed from ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 7 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:57.687570 29728 raft_consensus.cc:1261] T fc312e7affec4f67ada04db849694bdc P fdfeac4190e045f290e003b2298d988d [term 9 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20240501 20:01:57.689366 31445 consensus_queue.cc:1035] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:01:57.707146 29658 raft_consensus.cc:3037] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [term 8 FOLLOWER]: Advancing to term 9
I20240501 20:01:57.723178 29658 raft_consensus.cc:1261] T fc312e7affec4f67ada04db849694bdc P 4c8da1d580614449a636b855bf11b95f [term 9 FOLLOWER]: Refusing update from remote peer ee0d97247d394a8facfa774f15f3a67d: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
@ 0x7fe45dbc5bda std::__1::function<>::operator()() at ??:0
I20240501 20:01:57.729297 31791 consensus_queue.cc:1035] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 20:01:57.731834 29772 consensus_peers.cc:485] T fc312e7affec4f67ada04db849694bdc P ee0d97247d394a8facfa774f15f3a67d -> Peer 888d99522e4844118f8f1cf26f268c8c (127.27.217.196:37227): Couldn't send request to peer 888d99522e4844118f8f1cf26f268c8c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
@ 0x7fe454951237 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe454954692 kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
I20240501 20:01:57.791311 31920 raft_consensus.cc:979] T 84fa28fb55f44b61ba1a88ed775a78ba P 4c8da1d580614449a636b855bf11b95f: : Instructing follower fdfeac4190e045f290e003b2298d988d to start an election
I20240501 20:01:57.791826 31920 raft_consensus.cc:1067] T 84fa28fb55f44b61ba1a88ed775a78ba P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Signalling peer fdfeac4190e045f290e003b2298d988d to start an election
I20240501 20:01:57.793383 29728 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "84fa28fb55f44b61ba1a88ed775a78ba"
dest_uuid: "fdfeac4190e045f290e003b2298d988d"
from {username='slave'} at 127.0.0.1:36270
I20240501 20:01:57.793958 29728 raft_consensus.cc:482] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:01:57.794307 29728 raft_consensus.cc:3037] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:57.811694 29728 raft_consensus.cc:504] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:57.814152 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84fa28fb55f44b61ba1a88ed775a78ba" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:57.814735 29659 raft_consensus.cc:3032] T 84fa28fb55f44b61ba1a88ed775a78ba P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:01:57.815019 29659 raft_consensus.cc:729] T 84fa28fb55f44b61ba1a88ed775a78ba P 4c8da1d580614449a636b855bf11b95f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:57.815847 29659 consensus_queue.cc:260] T 84fa28fb55f44b61ba1a88ed775a78ba P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:57.816964 29659 raft_consensus.cc:3037] T 84fa28fb55f44b61ba1a88ed775a78ba P 4c8da1d580614449a636b855bf11b95f [term 1 FOLLOWER]: Advancing to term 2
@ 0x7fe45495464a _ZNSt3__18__invokeIRZN4kudu10ThreadPool12CreateThreadEvE3$_2JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS5_DpOS6_ at ??:0
I20240501 20:01:57.836128 31904 raft_consensus.cc:979] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d: : Instructing follower 4c8da1d580614449a636b855bf11b95f to start an election
I20240501 20:01:57.836704 31791 raft_consensus.cc:1067] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d [term 3 LEADER]: Signalling peer 4c8da1d580614449a636b855bf11b95f to start an election
I20240501 20:01:57.837815 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84fa28fb55f44b61ba1a88ed775a78ba" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:57.838357 29809 raft_consensus.cc:3037] T 84fa28fb55f44b61ba1a88ed775a78ba P ee0d97247d394a8facfa774f15f3a67d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:01:57.836444 29728 leader_election.cc:288] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:57.844489 31925 raft_consensus.cc:979] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d: : Instructing follower ee0d97247d394a8facfa774f15f3a67d to start an election
I20240501 20:01:57.845103 31798 raft_consensus.cc:1067] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Signalling peer ee0d97247d394a8facfa774f15f3a67d to start an election
I20240501 20:01:57.845469 29651 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aa8cd40cb4044d18a86001ce6ddf97ab"
dest_uuid: "4c8da1d580614449a636b855bf11b95f"
from {username='slave'} at 127.0.0.1:59136
I20240501 20:01:57.846009 29651 raft_consensus.cc:482] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:01:57.846326 29651 raft_consensus.cc:3037] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:57.846623 29800 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "211e27a3b0084b89a6dd7550849b5a65"
dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
from {username='slave'} at 127.0.0.1:41864
I20240501 20:01:57.847126 29800 raft_consensus.cc:482] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:01:57.847460 29800 raft_consensus.cc:3037] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:57.848210 29659 raft_consensus.cc:2454] T 84fa28fb55f44b61ba1a88ed775a78ba P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 2.
I20240501 20:01:57.850935 31920 raft_consensus.cc:979] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f: : Instructing follower fdfeac4190e045f290e003b2298d988d to start an election
I20240501 20:01:57.851352 31920 raft_consensus.cc:1067] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [term 5 LEADER]: Signalling peer fdfeac4190e045f290e003b2298d988d to start an election
I20240501 20:01:57.851914 29698 leader_election.cc:302] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:57.852769 31798 raft_consensus.cc:2781] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:01:57.853212 31798 raft_consensus.cc:686] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d [term 2 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:57.853314 29727 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9f261d2a05bf4c838532049601694b4a"
dest_uuid: "fdfeac4190e045f290e003b2298d988d"
from {username='slave'} at 127.0.0.1:36270
I20240501 20:01:57.853852 29727 raft_consensus.cc:482] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:01:57.854177 29727 raft_consensus.cc:3037] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:57.854051 31798 consensus_queue.cc:237] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:57.859090 29651 raft_consensus.cc:504] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:57.860397 29727 raft_consensus.cc:504] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:57.862135 29727 leader_election.cc:288] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:57.863222 29802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f261d2a05bf4c838532049601694b4a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:57.863770 29802 raft_consensus.cc:3037] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:57.847424 29809 raft_consensus.cc:2454] T 84fa28fb55f44b61ba1a88ed775a78ba P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 2.
I20240501 20:01:57.871513 29659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f261d2a05bf4c838532049601694b4a" candidate_uuid: "fdfeac4190e045f290e003b2298d988d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "4c8da1d580614449a636b855bf11b95f"
@ 0x7fe4549545da std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
I20240501 20:01:57.911365 29800 raft_consensus.cc:504] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
W20240501 20:01:57.913175 29700 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 46581.7us
I20240501 20:01:57.915948 29651 leader_election.cc:288] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 election: Requested vote from peers fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907), ee0d97247d394a8facfa774f15f3a67d (127.27.217.195:35861)
I20240501 20:01:57.916586 29727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8cd40cb4044d18a86001ce6ddf97ab" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:57.917268 29727 raft_consensus.cc:3037] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:57.919407 29658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211e27a3b0084b89a6dd7550849b5a65" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "4c8da1d580614449a636b855bf11b95f"
I20240501 20:01:57.919984 29658 raft_consensus.cc:3037] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:57.921177 29728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "211e27a3b0084b89a6dd7550849b5a65" candidate_uuid: "ee0d97247d394a8facfa774f15f3a67d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "fdfeac4190e045f290e003b2298d988d"
I20240501 20:01:57.921685 29728 raft_consensus.cc:3032] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Stepping down as leader of term 4
I20240501 20:01:57.922008 29728 raft_consensus.cc:729] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:57.922711 29728 consensus_queue.cc:260] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:57.923691 29728 raft_consensus.cc:3037] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:01:57.926294 29658 raft_consensus.cc:2454] T 211e27a3b0084b89a6dd7550849b5a65 P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 5.
I20240501 20:01:57.928608 29802 raft_consensus.cc:2454] T 9f261d2a05bf4c838532049601694b4a P ee0d97247d394a8facfa774f15f3a67d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 6.
I20240501 20:01:57.913256 29800 leader_election.cc:288] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 election: Requested vote from peers 4c8da1d580614449a636b855bf11b95f (127.27.217.193:33959), fdfeac4190e045f290e003b2298d988d (127.27.217.194:32907)
I20240501 20:01:57.929644 29728 raft_consensus.cc:2454] T 211e27a3b0084b89a6dd7550849b5a65 P fdfeac4190e045f290e003b2298d988d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee0d97247d394a8facfa774f15f3a67d in term 5.
I20240501 20:01:57.927955 29556 catalog_manager.cc:5484] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 1 to 2, leader changed from 4c8da1d580614449a636b855bf11b95f (127.27.217.193) to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 2 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:57.932366 29772 leader_election.cc:302] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, ee0d97247d394a8facfa774f15f3a67d; no voters:
I20240501 20:01:57.935672 29727 raft_consensus.cc:2454] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 4.
I20240501 20:01:57.936245 31904 raft_consensus.cc:2781] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:01:57.936726 31904 raft_consensus.cc:686] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [term 5 LEADER]: Becoming Leader. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:57.937075 29624 leader_election.cc:302] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8da1d580614449a636b855bf11b95f, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:57.937633 31904 consensus_queue.cc:237] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:57.940222 31862 raft_consensus.cc:2781] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:01:57.940532 29659 raft_consensus.cc:3032] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [term 5 LEADER]: Stepping down as leader of term 5
I20240501 20:01:57.940680 31862 raft_consensus.cc:686] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f [term 4 LEADER]: Becoming Leader. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:57.940892 29659 raft_consensus.cc:729] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4c8da1d580614449a636b855bf11b95f, State: Running, Role: LEADER
I20240501 20:01:57.941640 31862 consensus_queue.cc:237] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:57.941637 29659 consensus_queue.cc:260] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:57.943542 29659 raft_consensus.cc:3037] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:01:57.958184 29802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8cd40cb4044d18a86001ce6ddf97ab" candidate_uuid: "4c8da1d580614449a636b855bf11b95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "ee0d97247d394a8facfa774f15f3a67d"
I20240501 20:01:57.959249 29700 leader_election.cc:302] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee0d97247d394a8facfa774f15f3a67d, fdfeac4190e045f290e003b2298d988d; no voters:
I20240501 20:01:57.968156 29802 raft_consensus.cc:3032] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:01:57.970137 29802 raft_consensus.cc:729] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ee0d97247d394a8facfa774f15f3a67d, State: Running, Role: LEADER
I20240501 20:01:57.974885 29802 consensus_queue.cc:260] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
I20240501 20:01:57.979851 29659 raft_consensus.cc:2454] T 9f261d2a05bf4c838532049601694b4a P 4c8da1d580614449a636b855bf11b95f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfeac4190e045f290e003b2298d988d in term 6.
I20240501 20:01:57.984498 29802 raft_consensus.cc:3037] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:01:57.992501 29802 raft_consensus.cc:2454] T aa8cd40cb4044d18a86001ce6ddf97ab P ee0d97247d394a8facfa774f15f3a67d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8da1d580614449a636b855bf11b95f in term 4.
I20240501 20:01:57.979442 31925 raft_consensus.cc:2781] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [term 6 FOLLOWER]: Leader election won for term 6
I20240501 20:01:58.078385 29556 catalog_manager.cc:5484] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f reported cstate change: term changed from 3 to 4, leader changed from ee0d97247d394a8facfa774f15f3a67d (127.27.217.195) to 4c8da1d580614449a636b855bf11b95f (127.27.217.193). New cstate: current_term: 4 leader_uuid: "4c8da1d580614449a636b855bf11b95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:58.076277 31925 raft_consensus.cc:686] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [term 6 LEADER]: Becoming Leader. State: Replica: fdfeac4190e045f290e003b2298d988d, State: Running, Role: LEADER
I20240501 20:01:58.101635 29557 catalog_manager.cc:5484] T 211e27a3b0084b89a6dd7550849b5a65 P ee0d97247d394a8facfa774f15f3a67d reported cstate change: term changed from 4 to 5, leader changed from fdfeac4190e045f290e003b2298d988d (127.27.217.194) to ee0d97247d394a8facfa774f15f3a67d (127.27.217.195). New cstate: current_term: 5 leader_uuid: "ee0d97247d394a8facfa774f15f3a67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: HEALTHY } } }
I20240501 20:01:58.112740 31925 consensus_queue.cc:237] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } }
@ 0x7fe4549545a2 std::__1::__function::__alloc_func<>::operator()() at ??:0
W20240501 20:01:58.171394 29625 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.217.195:35861, user_credentials={real_user=slave}} blocked reactor thread for 73800.1us
I20240501 20:01:58.171696 29728 raft_consensus.cc:1261] T 77be7101c0fe4085b6706296f609870a P fdfeac4190e045f290e003b2298d988d [term 7 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240501 20:01:58.188699 31862 consensus_queue.cc:1035] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:01:58.230057 29556 catalog_manager.cc:5484] T 9f261d2a05bf4c838532049601694b4a P fdfeac4190e045f290e003b2298d988d reported cstate change: term changed from 5 to 6, leader changed from 4c8da1d580614449a636b855bf11b95f (127.27.217.193) to fdfeac4190e045f290e003b2298d988d (127.27.217.194). New cstate: current_term: 6 leader_uuid: "fdfeac4190e045f290e003b2298d988d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:58.247081 29802 raft_consensus.cc:1261] T 77be7101c0fe4085b6706296f609870a P ee0d97247d394a8facfa774f15f3a67d [term 7 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240501 20:01:58.253890 31966 consensus_queue.cc:1035] T 77be7101c0fe4085b6706296f609870a P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240501 20:01:58.273561 29809 raft_consensus.cc:1261] T 84fa28fb55f44b61ba1a88ed775a78ba P ee0d97247d394a8facfa774f15f3a67d [term 2 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
@ 0x7fe45495389e std::__1::__function::__func<>::operator()() at ??:0
I20240501 20:01:58.275596 31962 consensus_queue.cc:1035] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee0d97247d394a8facfa774f15f3a67d" member_type: VOTER last_known_addr { host: "127.27.217.195" port: 35861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:01:58.327966 29651 raft_consensus.cc:1261] T 84fa28fb55f44b61ba1a88ed775a78ba P 4c8da1d580614449a636b855bf11b95f [term 2 FOLLOWER]: Refusing update from remote peer fdfeac4190e045f290e003b2298d988d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:01:58.332448 31908 consensus_queue.cc:1035] T 84fa28fb55f44b61ba1a88ed775a78ba P fdfeac4190e045f290e003b2298d988d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8da1d580614449a636b855bf11b95f" member_type: VOTER last_known_addr { host: "127.27.217.193" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
@ 0x7fe45dbc5da5 std::__1::__function::__value_func<>::operator()() at ??:0
@ 0x7fe45dbc5bda std::__1::function<>::operator()() at ??:0
I20240501 20:01:58.393028 29729 raft_consensus.cc:1261] T aa8cd40cb4044d18a86001ce6ddf97ab P fdfeac4190e045f290e003b2298d988d [term 4 FOLLOWER]: Refusing update from remote peer 4c8da1d580614449a636b855bf11b95f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240501 20:01:58.395853 31966 consensus_queue.cc:1035] T aa8cd40cb4044d18a86001ce6ddf97ab P 4c8da1d580614449a636b855bf11b95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfeac4190e045f290e003b2298d988d" member_type: VOTER last_known_addr { host: "127.27.217.194" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
@ 0x7fe454933937 kudu::Thread::SuperviseThread() at ??:0